---
title: "The Dog the Cat Chased Stumped the Model: Measuring When"
description: "The authors build CenterBench to separate the two. 43 references. Among them: 0."
authors: ["Sangmitra Madhusudan", "Kaige Chen", "Ali Emami"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.19/"
doi: "10.18653/v1/2026.eacl-long.19"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/madhusudan-2026-dog-cat-chased-stumped-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/madhusudan-2026-dog-cat-chased-stumped-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts

# The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts

**Authors:** [Sangmitra Madhusudan](https://md.research.elamur.ai/free/en/authors/sangmitra-madhusudan/), Kaige Chen and [Ali Emami](https://md.research.elamur.ai/free/en/authors/ali-emami/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.19/) · [PDF](https://aclanthology.org/2026.eacl-long.19.pdf) · [DOI 10.18653/v1/2026.eacl-long.19](https://doi.org/10.18653/v1/2026.eacl-long.19) · [OpenAlex](https://openalex.org/W7140092266)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
When a language model correctly answers questions about a sentence with nested relative clauses, it is hard to tell whether it tracked the syntax or simply guessed from familiar real-world associations. The authors build CenterBench to separate the two. It contains 360 center-embedded sentences at nesting depths 1–6. Each plausible sentence (e.g., dogs chasing cats) has a structurally identical implausible twin in which entities perform actions typical of other entities, such as a mailman prescribing medicine. Every sentence comes with six types of comprehension questions, ranging from simple action lookup to multi-step causal tracing. If a model really relied on structure, its accuracy should be the same on both versions.

Across six models, accuracy falls as nesting deepens, and the advantage for plausible sentences grows steadily with depth. This pinpoints where models stop analyzing structure and start leaning on semantics. Humans show no such consistent pattern. The benchmark is therefore a diagnostic for settings where semantic cues may be misleading, such as figurative, historical or adversarial text.

## Key points
- **Dataset size and layout:** CenterBench has 9,720 questions, split evenly into 4,860 plausible and 4,860 implausible items. They come from 30 sentences per complexity level per subset.
- **How the sentences were built:**
  - Plausible sentences were generated with GPT-4-0613 from noun lists based on the Van Overschelde et al. (2004) norms, covering animals, occupations and vehicles.
  - Implausible sentences were made by circular verb swapping, using entity-specific verbs produced with claude-sonnet-4-20250514.
  - All sentences were checked by hand for temporal, semantic and syntactic validity.
- **Question types:** There are six, grouped into three tiers:
  - Easy: Action Performed and Agent Identification.
  - Medium: Entity Count and Nested Dependency.
  - Hard: Causal Sequence and Chain Consequence.
- **Main findings:**
  - Median gaps between plausible and implausible sentences reach as high as 26.8 percentage points.
  - On questions about resulting actions, plausibility actually lowers accuracy.
- **Reasoning models and humans:**
  - Reasoning models score higher, yet inspecting their reasoning reveals that they sometimes fall back on meaning-based shortcuts, reason excessively, or decline to state implausible relations.
  - Humans were tested up to level 4 and showed inconsistent plausibility effects.
  - The data and code are public on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Emory University](https://md.research.elamur.ai/free/en/universities/emory-university/) — University, Atlanta, United States · 36 works · 39,779 citations
- [Brock University](https://md.research.elamur.ai/free/en/universities/brock-university/) — University, St. Catharines, Canada · 3 works

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Sangmitra Madhusudan, Kaige Chen and Ali Emami. 2026. The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{madhusudan2026dog,
  title = {{The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts}},
  author = {Sangmitra Madhusudan, Kaige Chen and Ali Emami},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.19},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.19}
}
```

- [← previous The Displacement-Velocity Dissociation in Sign Language Learning: Kinematic Signatures of Event Structure in Novice ÖGS Signers](https://md.research.elamur.ai/free/en/papers/malaia-2026-displacement-velocity-dissociation-sign-language/)
- [next → The Double Bind: Revisiting Preprinting and Peer Review Two Years After the Removal of the ACL Anonymity Period](https://md.research.elamur.ai/free/en/papers/pranav-2026-double-bind-revisiting-preprinting-peer/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2025. Claude 3.7 sonnet. [https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet). [Accessed 15-07-2025]. [↗](https://www.anthropic.com/news/claude-3-7-sonnet)
2. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. [Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above](https://md.research.elamur.ai/free/en/papers/balepur-2025-which-these-best-describes-multiple/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
3. Nathaniel Berger, Stefan Riezler, Sebastian Ebert, and Artem Sokolov. 2021. Don’t search for a search method — simple heuristics suffice for adversarial text attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8216–8224, Online and Punta Cana, Do… [find](https://scholar.google.com/scholar?q=Nathaniel+Berger%2C+Stefan+Riezler%2C+Sebastian+Ebert%2C+and+Artem+Sokolov%2E+2021%2E+Don%E2%80%99t+search+for+a+search+method+%E2%80%94+simple+heuristics+suffice+for+adversarial+text+attacks%2E+In+Proceedings+of+the+2021+Confer)
4. Iona Carslaw, Sivan Milton, Nicolas Navarre, Ciyang Qing, and Wataru Uegaki. 2025. [Automatic extraction of clausal embedding based on large-scale english text data](https://md.research.elamur.ai/free/en/papers/carslaw-2025-automatic-extraction-clausal-embedding-based/). Preprint, arXiv:2506.14064.
5. Noam Chomsky and George A. Miller. 1963. Introduction to the formal analysis of natural languages. Journal of Symbolic Logic, 33(2):299–300. [find](https://scholar.google.com/scholar?q=Noam+Chomsky+and+George+A%2E+Miller%2E+1963%2E+Introduction+to+the+formal+analysis+of+natural+languages%2E+Journal+of+Symbolic+Logic%2C+33%282%29%3A299%E2%80%93300%2E)
6. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025a. [Deepseek-r1: Incentivizing reasoning capability in l…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025b. Deepseek-v3 technical report. Preprint, arXiv:… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
8. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. [Faith and fate: Limits of transformers on composition…](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/)
9. Benjamin Estermann, Luca A. Lanzendörfer, Yannick Niedermayr, and Roger Wattenhofer. 2024. Puzzles: A benchmark for neural algorithmic reasoning. Preprint, arXiv:2407.00401. — [arXiv:2407.00401](https://arxiv.org/abs/2407.00401) [find](https://scholar.google.com/scholar?q=Benjamin+Estermann%2C+Luca+A%2E+Lanzend%C3%B6rfer%2C+Yannick+Niedermayr%2C+and+Roger+Wattenhofer%2E+2024%2E+Puzzles%3A+A+benchmark+for+neural+algorithmic+reasoning%2E+Preprint%2C+arXiv%3A2407%2E00401%2E+%E2%80%94+%5BarXiv%3A2407%2E00401%5D%28https)
10. Fernanda Ferreira, Karl GD Bailey, and Vittoria Ferraro. 2002. Good-enough representations in language comprehension. Current directions in psychological science, 11(1):11–15. [find](https://scholar.google.com/scholar?q=Fernanda+Ferreira%2C+Karl+GD+Bailey%2C+and+Vittoria+Ferraro%2E+2002%2E+Good%2Denough+representations+in+language+comprehension%2E+Current+directions+in+psychological+science%2C+11%281%29%3A11%E2%80%9315%2E)
11. Candice Frances. 2024. Good enough processing: what have we learned in the 20 years since ferreira et al.(2002)? Frontiers in Psychology, 15:1323700. [find](https://scholar.google.com/scholar?q=Candice+Frances%2E+2024%2E+Good+enough+processing%3A+what+have+we+learned+in+the+20+years+since+ferreira+et+al%2E%282002%29%3F+Frontiers+in+Psychology%2C+15%3A1323700%2E)
12. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. [SyntaxGym: An online platform for targeted evaluation of language models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for…
13. E Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68(1):1–76. [find](https://scholar.google.com/scholar?q=E+Gibson%2E+1998%2E+Linguistic+complexity%3A+locality+of+syntactic+dependencies%2E+Cognition%2C+68%281%29%3A1%E2%80%9376%2E)
14. Daniel Hardt. 2025. [Sparks of pure competence in llms: the case of syntactic center embedding in english](https://md.research.elamur.ai/free/en/papers/hardt-2025-sparks-pure-competence-llms-case/). Society for Computation in Linguistics, 8(1).
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Preprint, arXiv:2009.03300.
16. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, Adriane Boyd, and 1 others. 2020. spacy: Industrial-strength natural language processing in python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+Adriane+Boyd%2C+and+1+others%2E+2020%2E+spacy%3A+Industrial%2Dstrength+natural+language+processing+in+python%2E)
17. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
18. Wenyue Hua, Tyler Wong, Fei Sun, Liangming Pan, Adam Jardine, and William Yang Wang. 2025. [InductionBench: LLMs fail in the simplest complexity class](https://md.research.elamur.ai/free/en/papers/hua-2025-inductionbench-llms-fail-simplest-complexity/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26526–26546, Vienna, Austri…
19. Richard Hudson. 1996. The difficulty of (so-called) self-embedded structures. Work. Pap. Linguist, 8:283– 314. [find](https://scholar.google.com/scholar?q=Richard+Hudson%2E+1996%2E+The+difficulty+of+%28so%2Dcalled%29+self%2Dembedded+structures%2E+Work%2E+Pap%2E+Linguist%2C+8%3A283%E2%80%93+314%2E)
20. Liwei Kang, Zirui Zhao, David Hsu, and Wee Sun Lee. 2024. On the empirical complexity of reasoning and planning in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2897–2936, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Liwei+Kang%2C+Zirui+Zhao%2C+David+Hsu%2C+and+Wee+Sun+Lee%2E+2024%2E+On+the+empirical+complexity+of+reasoning+and+planning+in+LLMs%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2024%2C+pages)
21. Fred Karlsson. 2007. Constraints on multiple center-embedding of clauses. Journal of Linguistics, 43(2):365–392. [find](https://scholar.google.com/scholar?q=Fred+Karlsson%2E+2007%2E+Constraints+on+multiple+center%2Dembedding+of+clauses%2E+Journal+of+Linguistics%2C+43%282%29%3A365%E2%80%93392%2E)
22. Koray Kavukcuoglu. 2025. Gemini 2.5: Our most intelligent ai model. [find](https://scholar.google.com/scholar?q=Koray+Kavukcuoglu%2E+2025%2E+Gemini+2%2E5%3A+Our+most+intelligent+ai+model%2E)
23. Yair Lakretz, Théo Desbordes, Dieuwke Hupkes, and Stanislas Dehaene. 2022. Can transformers process recursive nested constructions, like humans? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3226–3232, Gyeongju, Republic of Korea. International Committee on… [find](https://scholar.google.com/scholar?q=Yair+Lakretz%2C+Th%C3%A9o+Desbordes%2C+Dieuwke+Hupkes%2C+and+Stanislas+Dehaene%2E+2022%2E+Can+transformers+process+recursive+nested+constructions%2C+like+humans%3F+In+Proceedings+of+the+29th+International+Conference+on+)
24. Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition, 213:104699. [find](https://scholar.google.com/scholar?q=Yair+Lakretz%2C+Dieuwke+Hupkes%2C+Alessandra+Vergallito%2C+Marco+Marelli%2C+Marco+Baroni%2C+and+Stanislas+Dehaene%2E+2021%2E+Mechanisms+for+handling+nested+dependencies+in+neural%2Dnetwork+language+models+and+humans%2E)
25. Maria Lapata, Scott McDonald, and Frank Keller. 1999. Determinants of adjective-noun plausibility. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 30–36, Bergen, Norway. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maria+Lapata%2C+Scott+McDonald%2C+and+Frank+Keller%2E+1999%2E+Determinants+of+adjective%2Dnoun+plausibility%2E+In+Ninth+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+3)
26. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. [Entity-based knowledge conflicts in question answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Re…
27. Sangmitra Madhusudan, Robert Morabito, Skye Reid, Nikta Gohari Sadr, and Ali Emami. 2025. Fine-tuned LLMs are “time capsules” for tracking societal bias through books. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=Sangmitra+Madhusudan%2C+Robert+Morabito%2C+Skye+Reid%2C+Nikta+Gohari+Sadr%2C+and+Ali+Emami%2E+2025%2E+Fine%2Dtuned+LLMs+are+%E2%80%9Ctime+capsules%E2%80%9D+for+tracking+societal+bias+through+books%2E+In+Proceedings+of+the+2025+Confe)
28. George A. Miller and Noam Chomsky. 1963. Finitary models of language users. In D. Luce, editor, Handbook of Mathematical Psychology, pages 2–419. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller+and+Noam+Chomsky%2E+1963%2E+Finitary+models+of+language+users%2E+In+D%2E+Luce%2C+editor%2C+Handbook+of+Mathematical+Psychology%2C+pages+2%E2%80%93419%2E+John+Wiley+%26+Sons%2E)
29. Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, and Robert West. 2024. A glitch in the matrix? locating and detecting language model grounding with fakepedia. In Proceedings of the 62nd Annual Meeting of the Association for… [find](https://scholar.google.com/scholar?q=Giovanni+Monea%2C+Maxime+Peyrard%2C+Martin+Josifoski%2C+Vishrav+Chaudhary%2C+Jason+Eisner%2C+Emre+Kiciman%2C+Hamid+Palangi%2C+Barun+Patra%2C+and+Robert+West%2E+2024%2E+A+glitch+in+the+matrix%3F+locating+and+detecting+langu)
30. John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020. [Reevaluating adversarial examples in natural language](https://md.research.elamur.ai/free/en/papers/morris-2020-reevaluating-adversarial-examples-natural-language/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3829–3839, Online. Association for Computational Linguistics.
31. Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. 2024. In-context learning generalizes, but not always robustly: The case of syntax. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1… [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Albert+Webson%2C+Jackson+Petty%2C+and+Tal+Linzen%2E+2024%2E+In%2Dcontext+learning+generalizes%2C+but+not+always+robustly%3A+The+case+of+syntax%2E+In+Proceedings+of+the+2024+Conference+of+the+North+Amer)
32. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. [Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering](https://md.research.elamur.ai/free/en/papers/neeman-2023-disentqa-disentangling-parametric-contextual-knowledge/). Preprint, arXiv:2211.05655.
33. Shramay Palta, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, and Rachel Rudinger. 2024. Plausibly problematic questions in multiple-choice benchmarks for commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3451–3473, Miami, Florida… [find](https://scholar.google.com/scholar?q=Shramay+Palta%2C+Nishant+Balepur%2C+Peter+Rankel%2C+Sarah+Wiegreffe%2C+Marine+Carpuat%2C+and+Rachel+Rudinger%2E+2024%2E+Plausibly+problematic+questions+in+multiple%2Dchoice+benchmarks+for+commonsense+reasoning%2E+In+Fi)
34. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, and Mantas Mazeika et al. 2025. [Humanity’s last exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/). Pr…
35. Zhenting Qi, Hongyin Luo, Xuliang Huang, Zhuokai Zhao, Yibo Jiang, Xiangjun Fan, Himabindu Lakkaraju, and James Glass. 2024. Quantifying generalization complexity for large language models. Preprint, arXiv:2410.01769. — [arXiv:2410.01769](https://arxiv.org/abs/2410.01769) [find](https://scholar.google.com/scholar?q=Zhenting+Qi%2C+Hongyin+Luo%2C+Xuliang+Huang%2C+Zhuokai+Zhao%2C+Yibo+Jiang%2C+Xiangjun+Fan%2C+Himabindu+Lakkaraju%2C+and+James+Glass%2E+2024%2E+Quantifying+generalization+complexity+for+large+language+models%2E+Preprint%2C+)
36. Ethan Wilcox, Pranali Vani, and Roger Levy. 2021. A targeted assessment of incremental processing in neural language models and humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin… [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Pranali+Vani%2C+and+Roger+Levy%2E+2021%2E+A+targeted+assessment+of+incremental+processing+in+neural+language+models+and+humans%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for)
37. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American… [find](https://scholar.google.com/scholar?q=Zhaofeng+Wu%2C+Linlu+Qiu%2C+Alexis+Ross%2C+Ekin+Aky%C3%BCrek%2C+Boyuan+Chen%2C+Bailin+Wang%2C+Najoung+Kim%2C+Jacob+Andreas%2C+and+Yoon+Kim%2E+2024%2E+Reasoning+or+reciting%3F+exploring+the+capabilities+and+limitations+of+langua)
38. Linhao Zhang, Jintao Liu, Li Jin, Hao Wang, Kaiwen Wei, and Guangluan Xu. 2024. GOME: Grounding-based metaphor binding with conceptual elaboration for figurative language illustration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18500–18510, Miami… [find](https://scholar.google.com/scholar?q=Linhao+Zhang%2C+Jintao+Liu%2C+Li+Jin%2C+Hao+Wang%2C+Kaiwen+Wei%2C+and+Guangluan+Xu%2E+2024%2E+GOME%3A+Grounding%2Dbased+metaphor+binding+with+conceptual+elaboration+for+figurative+language+illustration%2E+In+Proceedings+)
39. Valentin Six, Evan Dufraisse, and Gaël de Chalendar. 2025. Decompositional reasoning for graph retrieval with large language models. Preprint, arXiv:2506.13380. — [arXiv:2506.13380](https://arxiv.org/abs/2506.13380) [find](https://scholar.google.com/scholar?q=Valentin+Six%2C+Evan+Dufraisse%2C+and+Ga%C3%ABl+de+Chalendar%2E+2025%2E+Decompositional+reasoning+for+graph+retrieval+with+large+language+models%2E+Preprint%2C+arXiv%3A2506%2E13380%2E+%E2%80%94+%5BarXiv%3A2506%2E13380%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
40. James P Van Overschelde, Katherine A Rawson, and John Dunlosky. 2004. Category norms: An updated and expanded version of the battig and montague (1969) norms. Journal of Memory and Language, 50(3):289–335. [find](https://scholar.google.com/scholar?q=James+P+Van+Overschelde%2C+Katherine+A+Rawson%2C+and+John+Dunlosky%2E+2004%2E+Category+norms%3A+An+updated+and+expanded+version+of+the+battig+and+montague+%281969%29+norms%2E+Journal+of+Memory+and+Language%2C+50%283%29%3A289)
41. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. [Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering](https://md.research.elamur.ai/free/en/papers/wallace-2019-trick-me-if-you-can/). Transactions of the Association for Computational Linguistics, 7:387– 401.
42. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Preprint, arXiv:2002.10957. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+2020%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+Preprint%2C+arXiv%3A2002%2E1095)
43. Ethan Wilcox, Roger Levy, and Richard Futrell. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 181–190, Florence, Italy. Association f… [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Roger+Levy%2C+and+Richard+Futrell%2E+2019%2E+Hierarchical+representation+in+neural+language+models%3A+Suppression+and+recovery+of+expectations%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNL)

*Updated 1 November 2026*