---
title: "TurBLiMP: A Turkish Benchmark of Linguistic — Başar 2025"
description: "The authors created TurBLiMP, the first set of linguistic minimal pairs for Turkish."
authors: ["Ezgi Başar", "Francesca Padovani", "Jaap Jumelet", "Arianna Bisazza"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.834/"
doi: "10.18653/v1/2025.emnlp-main.834"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ar-2025-turblimp-turkish-benchmark-linguistic-minimal/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ar-2025-turblimp-turkish-benchmark-linguistic-minimal/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2025](https://md.research.elamur.ai/free/papers/2025/page/2/) / TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

# TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

**Authors:** Ezgi Başar, [Francesca Padovani](https://md.research.elamur.ai/free/authors/francesca-padovani/), [Jaap Jumelet](https://md.research.elamur.ai/free/authors/jaap-jumelet/), [Arianna Bisazza](https://md.research.elamur.ai/free/authors/arianna-bisazza/) · **Year:** 2025 · **Venue:** Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.834/) · [PDF](https://aclanthology.org/2025.emnlp-main.834.pdf) · [DOI 10.18653/v1/2025.emnlp-main.834](https://doi.org/10.18653/v1/2025.emnlp-main.834) · [OpenAlex](https://openalex.org/W4416036196)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [14](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The authors created TurBLiMP, the first set of linguistic minimal pairs for Turkish. It makes it possible to test the grammatical knowledge of monolingual and multilingual language models. The benchmark covers 16 grammatical phenomena, from anaphor agreement and argument structure to NPIs, ellipsis and island constraints. Each phenomenon has 1000 pairs in which one sentence is acceptable and the other is not. The pairs were created with a hybrid method adapted to the specifics of Turkish. In addition, the authors designed experimental paradigms that vary word order and the type of subordinate construction. In Turkish, subordination is expressed mainly through suffixes. For each pair, the authors compared the log-probabilities of the two sentences in 13 models. The results were compared against newly collected acceptability judgments from native speakers. Even the strongest LLMs make errors where humans answer confidently. Models respond to non-canonical word order and complex morphology differently from humans. The work extends BLiMP-style testing to an agglutinative language with free word order. It can therefore serve as a model for other morphologically rich languages.

## Key points
- Size: 16 phenomena with 1000 pairs each, 16 000 in total. Some of the phenomena are specific to Turkish, such as suspended affixation. Each phenomenon has one paradigm, except Argument Structure, which has two: transitive and ditransitive.
- 13 models were tested. Larger models are generally more accurate, but there are exceptions. Much smaller monolingual models (in particular BERTurk) often outperform larger multilingual ones and are on par with the best of them.
- Ellipsis (backward gapping) remains difficult for models of every size and architecture. This is where model behavior diverges most noticeably from human judgments.
- The experimental paradigms test two parameters: word order (Turkish allows all six permutations of subject, object and verb) and different types of subordination (-DIK, finite, adverbial clauses). Even the best models lose accuracy on them, which the standard evaluation does not reveal.
- Instruct variants of the models were not evaluated because, for example, Gemma 3 (4B) Instruct is on average 4.4% worse than the base version. The data are open: github.com/ezgibasar/turblimp.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 12 not yet.

- [UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu](https://md.research.elamur.ai/free/papers/adeeba-2026-urblimp-benchmark-evaluating-linguistic-competence/)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://md.research.elamur.ai/free/papers/agarwal-2025-mechanisms-vs-outcomes-probing-syntax/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{basar2025turblimp,
  title = {{TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs}},
  author = {Başar, Ezgi and Padovani, Francesca and Jumelet, Jaap and Bisazza, Arianna},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.834},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.834}
}
```

- [← previous TriFine: A Large-Scale Dataset of Vision-Audio-Subtitle for Tri-Modal Machine Translation and Benchmark with Fine-Grained Annotated Tags](https://md.research.elamur.ai/free/papers/guan-2025-trifine-large-scale-dataset-vision/)
- [next → UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions](https://md.research.elamur.ai/free/papers/ud-2025-yang-ud-english-childes-collected-resource/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/papers/2025/page/2/)

## References

54 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmet Afşın Akın and Mehmet Dündar Akın. 2007. Zemberek, an open source nlp framework for turkic languages. Structure, 10:1–5. [find](https://scholar.google.com/scholar?q=Ahmet+Afs%CC%A7%C4%B1n+Ak%C4%B1n+and+Mehmet+D%C3%BCndar+Ak%C4%B1n%2E+2007%2E+Zemberek%2C+an+open+source+nlp+framework+for+turkic+languages%2E+Structure%2C+10%3A1%E2%80%935%2E)
2. Bjarki Ármannsson, Finnur Ágúst Ingimundarson, and Einar Freyr Sigurðsson. 2025. An Icelandic linguistic benchmark for large language models. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT… [find](https://scholar.google.com/scholar?q=Bjarki+%C3%81rmannsson%2C+Finnur+%C3%81g%C3%BAst+Ingimundarson%2C+and+Einar+Freyr+Sigur%C3%B0sson%2E+2025%2E+An+Icelandic+linguistic+benchmark+for+large+language+models%2E+In+Proceedings+of+the+Joint+25th+Nordic+Conference+on+Comp)
3. Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Catherine+Arnett+and+Benjamin+Bergen%2E+2025%2E+Why+do+language+models+perform+worse+for+morphologically+complex+languages%3F+In+Proceedings+of+the+31st+International+Conference+on+Computational+Linguistics)
4. Z. Ceyda Arslan-Kechriotis. 2009. Referentiality in turkish: NP/DP. In Essays on Turkish Linguistics: Proceedings of the 14th International Conference on Turkish Linguistics, 6-8 August 2008, pages 83–92, Wiesbaden. Harrassowitz Verlag. [find](https://scholar.google.com/scholar?q=Z%2E+Ceyda+Arslan%2DKechriotis%2E+2009%2E+Referentiality+in+turkish%3A+NP%2FDP%2E+In+Essays+on+Turkish+Linguistics%3A+Proceedings+of+the+14th+International+Conference+on+Turkish+Linguistics%2C+6%2D8+August+2008%2C+pages+83)
5. Duygu Ataman, Matteo Negri, Marco Turchi, and Marcello Federico. 2017. Linguistically motivated vocabulary reduction for neural machine translation from turkish to english. The Prague Bulletin of Mathematical Linguistics, 108:331 – 342. [find](https://scholar.google.com/scholar?q=Duygu+Ataman%2C+Matteo+Negri%2C+Marco+Turchi%2C+and+Marcello+Federico%2E+2017%2E+Linguistically+motivated+vocabulary+reduction+for+neural+machine+translation+from+turkish+to+english%2E+The+Prague+Bulletin+of+Math)
6. Cem Bozşahin. 2000. Gapping and word order in Turkish. In Proceedings of the 10th International Conference on Turkish Linguistics, pages 58–66. [find](https://scholar.google.com/scholar?q=Cem+Bozs%CC%A7ahin%2E+2000%2E+Gapping+and+word+order+in+Turkish%2E+In+Proceedings+of+the+10th+International+Conference+on+Turkish+Linguistics%2C+pages+58%E2%80%9366%2E)
7. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024. Goldfish: Monolingual language models for 350 languages. Preprint. [find](https://scholar.google.com/scholar?q=Tyler+A%2E+Chang%2C+Catherine+Arnett%2C+Zhuowen+Tu%2C+and+Benjamin+K%2E+Bergen%2E+2024%2E+Goldfish%3A+Monolingual+language+models+for+350+languages%2E+Preprint%2E)
8. Nadezhda Chirkova and Vassilina Nikoulina. 2024. Zero-shot cross-lingual transfer in instruction tuning of large language models. In Proceedings of the 17th International Natural Language Generation Conference, pages 695–708, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nadezhda+Chirkova+and+Vassilina+Nikoulina%2E+2024%2E+Zero%2Dshot+cross%2Dlingual+transfer+in+instruction+tuning+of+large+language+models%2E+In+Proceedings+of+the+17th+International+Natural+Language+Generation+C)
9. Noam Chomsky. 1965. Aspects of the Theory of Syntax, 50 edition. The MIT Press, Cambridge, MA, USA. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1965%2E+Aspects+of+the+Theory+of+Syntax%2C+50+edition%2E+The+MIT+Press%2C+Cambridge%2C+MA%2C+USA%2E)
10. Noam Chomsky. 1981. Lectures on Government and Binding. Number 9 in Studies in Generative Grammar. Foris Publications, Dordrecht. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1981%2E+Lectures+on+Government+and+Binding%2E+Number+9+in+Studies+in+Generative+Grammar%2E+Foris+Publications%2C+Dordrecht%2E)
11. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell%2C+Christo+Kirov%2C+John+Sylak%2DGlassman%2C+David+Yarowsky%2C+Jason+Eisner%2C+and+Mans+Hulden%2E+2016%2E)
12. The SIGMORPHON 2016 shared Task— Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+SIGMORPHON+2016+shared+Task%E2%80%94+Morphological+reinflection%2E+In+Proceedings+of+the+14th+SIGMORPHON+Workshop+on+Computational+Research+in+Phonetics%2C+Phonology%2C+and+Morphology%2C+pages+10%E2%80%9322%2C+Berlin%2C+Germ)
13. Fred J. Damerau. 1964. A technique for computer detection and correction of spelling errors. Commun. ACM, 7(3):171–176. [find](https://scholar.google.com/scholar?q=Fred+J%2E+Damerau%2E+1964%2E+A+technique+for+computer+detection+and+correction+of+spelling+errors%2E+Commun%2E+ACM%2C+7%283%29%3A171%E2%80%93176%2E)
14. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac,… [find](https://scholar.google.com/scholar?q=John+Dang%2C+Shivalika+Singh%2C+Daniel+D%E2%80%99souza%2C+Arash+Ahmadian%2C+Alejandro+Salamanca%2C+Madeline+Smith%2C+Aidan+Peppin%2C+Sungjin+Hong%2C+Manoj+Govindassamy%2C+Terrence+Zhao%2C+Sandra+Kublik%2C+Meor+Amer%2C+Viraat+Aryabum) [our card](https://md.research.elamur.ai/free/papers/dang-2024-aya-expanse-combining-research-breakthroughs/)
15. Mürvet Enç. 1991. The semantics of specificity. Linguistic Inquiry, 22(1):1–25. [find](https://scholar.google.com/scholar?q=M%C3%BCrvet+En%C3%A7%2E+1991%2E+The+semantics+of+specificity%2E+Linguistic+Inquiry%2C+22%281%29%3A1%E2%80%9325%2E)
16. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327, Brussels,… [find](https://scholar.google.com/scholar?q=Daniela+Gerz%2C+Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2018%2E+On+the+relation+between+linguistic+typology+and+%28limitations+of%29+multilingual+language+modeling%2E+In+Proceedings+o)
17. Aslı Göksel and Celia Kerslake. 2005. Turkish: A Comprehensive Grammar. Comprehensive grammars. Routledge. [find](https://scholar.google.com/scholar?q=Asl%C4%B1+G%C3%B6ksel+and+Celia+Kerslake%2E+2005%2E+Turkish%3A+A+Comprehensive+Grammar%2E+Comprehensive+grammars%2E+Routledge%2E)
18. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2274–2286, Bangkok, Thailand. Assoc… [find](https://scholar.google.com/scholar?q=Omer+Goldman%2C+Avi+Caciularu%2C+Matan+Eyal%2C+Kris+Cao%2C+Idan+Szpektor%2C+and+Reut+Tsarfaty%2E+2024%2E+Unpacking+tokenization%3A+Evaluating+text+compression+and+its+correlation+with+model+performance%2E+In+Findings+o)
19. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. 2024. Language models align with human judgments on key grammatical constructions. Proceedings of the National Academy of Sciences, 121(36):e2400917121. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Kyle+Mahowald%2C+Gary+Lupyan%2C+Anna+Ivanova%2C+and+Roger+Levy%2E+2024%2E+Language+models+align+with+human+judgments+on+key+grammatical+constructions%2E+Proceedings+of+the+National+Academy+of+Science)
20. Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, and Lonneke Van Der Plas. 2025. Evaluating morphological compositional generalization in large language models. In Proceedings of the 2025 Conference of the Nations of the Amer… [find](https://scholar.google.com/scholar?q=Mete+Ismayilzada%2C+Defne+Circi%2C+Jonne+S%C3%A4lev%C3%A4%2C+Hale+Sirin%2C+Abdullatif+K%C3%B6ksal%2C+Bhuwan+Dhingra%2C+Antoine+Bosselut%2C+Duygu+Ataman%2C+and+Lonneke+Van+Der+Plas%2E+2025%2E+Evaluating+morphological+compositional+gener)
21. Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of LSTMs to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222–231, Brussels, Belgium. Association for C… [find](https://scholar.google.com/scholar?q=Jaap+Jumelet+and+Dieuwke+Hupkes%2E+2018%2E+Do+language+models+understand+anything%3F+on+the+ability+of+LSTMs+to+understand+negative+polarity+items%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Ana)
22. Jaap Jumelet, Leonie Weissweiler, and Arianna Bisazza. 2025. Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768. [find](https://scholar.google.com/scholar?q=Jaap+Jumelet%2C+Leonie+Weissweiler%2C+and+Arianna+Bisazza%2E+2025%2E+Multiblimp+1%2E0%3A+A+massively+multilingual+benchmark+of+linguistic+minimal+pairs%2E+Preprint%2C+arXiv%3A2504%2E02768%2E)
23. Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925–935, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Carina+Kauf+and+Anna+Ivanova%2E+2023%2E+A+better+way+to+do+masked+language+model+scoring%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers)
24. H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Ege-men Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, and M. Fatih Amasyali. 2024. Introducing cosmosgpt: Monolingual training for turkish language models. arXiv preprint arXiv:2404.17336. — [arXiv:2404.17336](https://arxiv.org/abs/2404.17336) [find](https://scholar.google.com/scholar?q=H%2E+Toprak+Kesgin%2C+M%2E+Kaan+Yuce%2C+Eren+Dogan%2C+M%2E+Ege%2Dmen+Uzun%2C+Atahan+Uz%2C+H%2E+Emre+Seyrek%2C+Ahmed+Zeer%2C+and+M%2E+Fatih+Amasyali%2E+2024%2E+Introducing+cosmosgpt%3A+Monolingual+training+for+turkish+language+models)
25. Jaklin Kornfilt. 2003a. Scrambling, Subscrambling, and Case in Turkish, chapter 6. John Wiley & Sons, Ltd. [find](https://scholar.google.com/scholar?q=Jaklin+Kornfilt%2E+2003a%2E+Scrambling%2C+Subscrambling%2C+and+Case+in+Turkish%2C+chapter+6%2E+John+Wiley+%26+Sons%2C+Ltd%2E)
26. Jaklin Kornfilt. 2003b. Subject Case in Turkish nominalized clauses, pages 129–216. De Gruyter Mouton, Berlin, Boston. [find](https://scholar.google.com/scholar?q=Jaklin+Kornfilt%2E+2003b%2E+Subject+Case+in+Turkish+nominalized+clauses%2C+pages+129%E2%80%93216%2E+De+Gruyter+Mouton%2C+Berlin%2C+Boston%2E)
27. Daria Kryvosheieva and Roger Levy. 2025. Controlled evaluation of syntactic knowledge in multilingual language models. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 402–413, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daria+Kryvosheieva+and+Roger+Levy%2E+2025%2E+Controlled+evaluation+of+syntactic+knowledge+in+multilingual+language+models%2E+In+Proceedings+of+the+First+Workshop+on+Language+Models+for+Low%2DResource+Language)
28. Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023. Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models. Preprint, arXiv:2309.06085. [find](https://scholar.google.com/scholar?q=Wei+Qi+Leong%2C+Jian+Gang+Ngui%2C+Yosephine+Susanto%2C+Hamsawardhini+Rengarajan%2C+Kengatharaiyer+Sarveswaran%2C+and+William+Chandra+Tjhi%2E+2023%2E+Bhasa%3A+A+holistic+southeast+asian+linguistic+and+cultural+evaluat)
29. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+ability+of+LSTMs+to+learn+syntax%2Dsensitive+dependencies%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A521%E2%80%93535%2E) [our card](https://md.research.elamur.ai/free/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
30. Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, and Hai Hu. 2024. Zhoblimp: a systematic assessment of language models with linguistic minimal pairs in chinese. Preprint, arXiv:2411.06096. [find](https://scholar.google.com/scholar?q=Yikang+Liu%2C+Yeting+Shen%2C+Hongao+Zhu%2C+Lilong+Xu%2C+Zhiheng+Qian%2C+Siyuan+Song%2C+Kejia+Zhang%2C+Jialong+Tang%2C+Pei+Zhang%2C+Baosong+Yang%2C+Rui+Wang%2C+and+Hai+Hu%2E+2024%2E+Zhoblimp%3A+a+systematic+assessment+of+language)
31. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurollm: Multilingual language m… [find](https://scholar.google.com/scholar?q=Pedro+Henrique+Martins%2C+Patrick+Fernandes%2C+Jo%C3%A3o+Alves%2C+Nuno+M%2E+Guerreiro%2C+Ricardo+Rei%2C+Duarte+M%2E+Alves%2C+Jos%C3%A9+Pombal%2C+Amin+Farajian%2C+Manuel+Faysse%2C+Mateusz+Klimaszewski%2C+Pierre+Colombo%2C+Barry+Haddow%2C+J) [our card](https://md.research.elamur.ai/free/papers/martins-2024-eurollm-multilingual-language-models-europe/)
32. Meta. 2024. Llama 3.1: Open and efficient foundation language models. [Software]. Version 3.1. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+Llama+3%2E1%3A+Open+and+efficient+foundation+language+models%2E+%5BSoftware%5D%2E+Version+3%2E1%2E)
33. Kanishka Misra. 2022. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv preprint arXiv:2203.13112. [find](https://scholar.google.com/scholar?q=Kanishka+Misra%2E+2022%2E+minicons%3A+Enabling+flexible+behavioral+and+representational+analyses+of+transformer+language+models%2E+arXiv+preprint+arXiv%3A2203%2E13112%2E)
34. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Comput… [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Garrett+Nicolai%2C+Panayiota+Petrou%2DZeniou%2C+Natalia+Talmina%2C+and+Tal+Linzen%2E+2020%2E+Cross%2Dlinguistic+syntactic+evaluation+of+word+prediction+models%2E+In+Proceedings+of+the+58th+Annual+Meeti)
35. Mine Nakipoğlu, Berna A. Uzundağ, and F. Nihan Ke-trez. 2023. Analogy is indispensable but rule is a must: Insights from turkish. Journal of Child Language, 50(2):437–463. [find](https://scholar.google.com/scholar?q=Mine+Nakipog%CC%86lu%2C+Berna+A%2E+Uzundag%CC%86%2C+and+F%2E+Nihan+Ke%2Dtrez%2E+2023%2E+Analogy+is+indispensable+but+rule+is+a+must%3A+Insights+from+turkish%2E+Journal+of+Child+Language%2C+50%282%29%3A437%E2%80%93463%2E)
36. Hyunji Hayley Park, Katherine J. Zhang, Coleman Haley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261–276. [find](https://scholar.google.com/scholar?q=Hyunji+Hayley+Park%2C+Katherine+J%2E+Zhang%2C+Coleman+Haley%2C+Kenneth+Steimel%2C+Han+Liu%2C+and+Lane+Schwartz%2E+2021%2E+Morphology+matters%3A+A+multilingual+language+modeling+analysis%2E+Transactions+of+the+Association)
37. Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 959–968, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Pauls+and+Dan+Klein%2E+2012%2E+Large%2Dscale+syntactic+language+modeling+with+treelets%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers)
38. Stefan Schweter. 2020. Berturk - bert models for turkish. [find](https://scholar.google.com/scholar?q=Stefan+Schweter%2E+2020%2E+Berturk+%2D+bert+models+for+turkish%2E)
39. Kutay Serova. 2019. Head movement, suspended affixation, and the turkish clausal spine. Proceedings of the Workshop on Turkic and Languages in Contact with Turkic, 4:89. [find](https://scholar.google.com/scholar?q=Kutay+Serova%2E+2019%2E+Head+movement%2C+suspended+affixation%2C+and+the+turkish+clausal+spine%2E+Proceedings+of+the+Workshop+on+Turkic+and+Languages+in+Contact+with+Turkic%2C+4%3A89%2E)
40. Dan I. Slobin and Thomas G. Bever. 1982. Children use canonical sentence schemas: A crosslinguistic study of word order and inflections. Cognition, 12(3):229– 265. [find](https://scholar.google.com/scholar?q=Dan+I%2E+Slobin+and+Thomas+G%2E+Bever%2E+1982%2E+Children+use+canonical+sentence+schemas%3A+A+crosslinguistic+study+of+word+order+and+inflections%2E+Cognition%2C+12%283%29%3A229%E2%80%93+265%2E)
41. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Taiga+Someya+and+Yohei+Oseki%2E+2023%2E+JBLiMP%3A+Japanese+benchmark+of+linguistic+minimal+pairs%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2023%2C+pages+1581%E2%80%931594%2C+Dubrovnik%2C+Croatia)
42. Siyuan Song, Jennifer Hu, and Kyle Mahowald. 2025. Language models fail to introspect about their knowledge of language. Preprint, arXiv:2503.07513. [find](https://scholar.google.com/scholar?q=Siyuan+Song%2C+Jennifer+Hu%2C+and+Kyle+Mahowald%2E+2025%2E+Language+models+fail+to+introspect+about+their+knowledge+of+language%2E+Preprint%2C+arXiv%3A2503%2E07513%2E)
43. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, and Mohit Iyyer. 2022. SLING: Sino linguistic evaluation of large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606–4634, Abu Dhabi, United Arab Emirates. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Yixiao+Song%2C+Kalpesh+Krishna%2C+Rajesh+Bhatt%2C+and+Mohit+Iyyer%2E+2022%2E+SLING%3A+Sino+linguistic+evaluation+of+large+language+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Lan)
44. Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L. Frank. 2025. Blimp-nl: A corpus of dutch minimal pairs and acceptability judgments for language model evaluation. Computational Linguistics, pages 1–39. [find](https://scholar.google.com/scholar?q=Michelle+Suijkerbuijk%2C+Zo%C3%AB+Prins%2C+Marianne+de+Heer+Kloots%2C+Willem+Zuidema%2C+and+Stefan+L%2E+Frank%2E+2025%2E+Blimp%2Dnl%3A+A+corpus+of+dutch+minimal+pairs+and+acceptability+judgments+for+language+model+evaluatio)
45. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Kon-cha, Alena Fenogenova, Ekaterina Artemova, and Vladislav Mikhailov. 2024. RuBLiMP: Russian benchmark of linguistic minimal pairs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9268–9299, Miami, Florid… [find](https://scholar.google.com/scholar?q=Ekaterina+Taktasheva%2C+Maxim+Bazhukov%2C+Kirill+Kon%2Dcha%2C+Alena+Fenogenova%2C+Ekaterina+Artemova%2C+and+Vladislav+Mikhailov%2E+2024%2E+RuBLiMP%3A+Russian+benchmark+of+linguistic+minimal+pairs%2E+In+Proceedings+of+the)
46. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+Louis+Rouillard%2C+Thomas+Mesnard%2C+Geoffrey)
47. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
48. Utku Türk, Furkan Atmaca, Şaziye Betül Özateş, Gözde Berk, Seyyit Talha Bedir, Abdullatif Köksal, Balkız Öztürk Başaran, Tunga Güngör, and Arzucan Özgür. 2022. Resources for turkish dependency parsing: introducing the boun treebank and the boat annotation tool. Lang. Resour. Eval., 56(1):259–307. [find](https://scholar.google.com/scholar?q=Utku+T%C3%BCrk%2C+Furkan+Atmaca%2C+S%CC%A7aziye+Bet%C3%BCl+%C3%96zates%CC%A7%2C+G%C3%B6zde+Berk%2C+Seyyit+Talha+Bedir%2C+Abdullatif+K%C3%B6ksal%2C+Balk%C4%B1z+%C3%96zt%C3%BCrk+Bas%CC%A7aran%2C+Tunga+G%C3%BCng%C3%B6r%2C+and+Arzucan+%C3%96zg%C3%BCr%2E+2022%2E+Resources+for+turkish+dependency+pars)
49. Héctor Javier Vázquez Martínez, Annika Heuser, Charles Yang, and Jordan Kodner. 2023. Evaluating neural language models as cognitive models of language acquisition. In Proceedings of the 1st Gen-Bench Workshop on (Benchmarking) Generalisation in NLP, pages 48–64, Singapore. Association for Computati… [find](https://scholar.google.com/scholar?q=H%C3%A9ctor+Javier+V%C3%A1zquez+Mart%C3%ADnez%2C+Annika+Heuser%2C+Charles+Yang%2C+and+Jordan+Kodner%2E+2023%2E+Evaluating+neural+language+models+as+cognitive+models+of+language+acquisition%2E+In+Proceedings+of+the+1st+Gen%2DBench)
50. Sinan Çakır. 2016. Island constraints and adjunct & argument asymmetry in turkish. Dilbilim Araştır-maları Dergisi, 27:1–1. [find](https://scholar.google.com/scholar?q=Sinan+%C3%87ak%C4%B1r%2E+2016%2E+Island+constraints+and+adjunct+%26+argument+asymmetry+in+turkish%2E+Dilbilim+Aras%CC%A7t%C4%B1r%2Dmalar%C4%B1+Dergisi%2C+27%3A1%E2%80%931%2E)
51. Sumru A. Özsoy. 2009. Argument structure, animacy, syntax and semantics of passivization in Turkish: A corpus-based approach, pages 259–279. John Benjamins Publishing Company. [find](https://scholar.google.com/scholar?q=Sumru+A%2E+%C3%96zsoy%2E+2009%2E+Argument+structure%2C+animacy%2C+syntax+and+semantics+of+passivization+in+Turkish%3A+A+corpus%2Dbased+approach%2C+pages+259%E2%80%93279%2E+John+Benjamins+Publishing+Company%2E)
52. A Our experimental paradigms only target minimal pairs belonging to Argument Structure Transitive and Argument Structure Ditransitive phenomena. Below is an example of the transitive baseline. (3) [find](https://scholar.google.com/scholar?q=A+Our+experimental+paradigms+only+target+minimal+pairs+belonging+to+Argument+Structure+Transitive+and+Argument+Structure+Ditransitive+phenomena%2E+Below+is+an+example+of+the+transitive+baseline%2E+%283%29)
53. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo… [find](https://scholar.google.com/scholar?q=Beilei+Xiang%2C+Changbing+Yang%2C+Yu+Li%2C+Alex+Warstadt%2C+and+Katharina+Kann%2E+2021%2E+CLiMP%3A+A+benchmark+for+Chinese+language+model+evaluation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of)
54. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2025. Qwen2.5 technical report. arXiv preprint arXi… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)