---
title: "Minimal Pair-Based Evaluation of Code-Switching — Sterner 2025"
description: "Large language models can already produce mixed-language text, but there has been no scalable… 50 references, cited by 6 papers here."
authors: ["Igor Sterner", "Simone Teufel"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.910/"
doi: "10.18653/v1/2025.acl-long.910"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sterner-2025-minimal-pair-based-evaluation-code/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sterner-2025-minimal-pair-based-evaluation-code/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Minimal Pair-Based Evaluation of Code-Switching

# Minimal Pair-Based Evaluation of Code-Switching

**Authors:** [Igor Sterner](https://md.research.elamur.ai/free/en/authors/igor-sterner/) and [Simone Teufel](https://md.research.elamur.ai/free/en/authors/simone-teufel/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.910/) · [PDF](https://aclanthology.org/2025.acl-long.910.pdf) · [DOI 10.18653/v1/2025.acl-long.910](https://doi.org/10.18653/v1/2025.acl-long.910) · [OpenAlex](https://openalex.org/W4412944982)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/sterner-2025-minimal-pair-based-evaluation-code/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models can already produce mixed-language text, but there has been no scalable, language-independent way to check whether they mix languages the way bilinguals do. Earlier evaluations relied on absolute human ratings of individual sentences, which are subjective, hard to compare, and show low agreement. Sterner and Teufel borrow the minimal-pair approach from monolingual acceptability research, as in BLiMP. They take naturally occurring code-switched (CS) sentences, mostly from Twitter, and translate one word next to a switch point into the other language. This produces a contrasting variant with the same number of switch points. The resulting ACS benchmark covers 11 language pairs, with up to 1,000 pairs each. Bilingual judges consistently chose the original sentences, which confirms that each pair differs in acceptability. Among the LLMs tested, larger models more reliably gave higher probability to the authentic sentence, and Llama-3.1 405B performed best. Swapping closed-class words caused larger probability gaps than swapping open-class words, which matches theoretical predictions. Because human validation only has to be done once, the benchmark can track future models cheaply and also supports direct tests of linguistic hypotheses about CS.

## Key points
- **Data sources:**
  - German–English pairs come from 27.5M recent tweets from the authors' earlier corpus.
  - Twitter data from DeLucia et al. (2022) supplies eight further languages paired with English: Dutch, French, Italian, Spanish, Danish, Swedish, Indonesian, and Turkish.
  - Spoken corpora, such as Chinese–English from Lyu et al. (2010), are also included.
- **Processing pipeline:**
  - Token-level language ID uses the AnE system.
  - Lingua re-checks the language of each tweet.
  - Wiktionary lists are used to flag borrowings.
  - Sentence splitting uses the sat-12-sm segmentation model.
- **Human validation:** Bilinguals preferred the natural sentence for every language pair, with agreement of κ=0.57. Apart from German–English, only one participant per language pair was used.
- **Word-class effect:** For one language pair, the LLM probability margins were 4.3 for DET, 5.3 for ADP, 6.5 for PRON and 7.9 for AUX. The closed- vs open-class difference was significant for all pairs except German–English and Italian–English.
- **Imbalance and limitations:** In that experiment, closed-class changes were underrepresented (147 vs 573 pairs). Other limitations include noisy Twitter data and the possibility that the test data appeared in Llama-3 pre-training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 243 works · 121,396 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Igor Sterner and Simone Teufel. 2025. Minimal Pair-Based Evaluation of Code-Switching. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sterner2025minimal,
  title = {{Minimal Pair-Based Evaluation of Code-Switching}},
  author = {Igor Sterner and Simone Teufel},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.910},
  url = {https://doi.org/10.18653/v1/2025.acl-long.910}
}
```

- [← previous MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs](https://md.research.elamur.ai/free/en/papers/liu-2025-metafaith-faithful-natural-language-uncertainty/)
- [next → OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA](https://md.research.elamur.ai/free/en/papers/alam-2025-oasis-multilingual-multimodal-dataset-culturally/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

50 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI2. 2024. OLMo: Accelerating the science of language models. Preprint, arXiv:2402.00838v4. [find](https://scholar.google.com/scholar?q=AI2%2E+2024%2E+OLMo%3A+Accelerating+the+science+of+language+models%2E+Preprint%2C+arXiv%3A2402%2E00838v4%2E)
2. BigScience. 2023. BLOOM: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100v4. [find](https://scholar.google.com/scholar?q=BigScience%2E+2023%2E+BLOOM%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E+Preprint%2C+arXiv%3A2211%2E05100v4%2E)
3. Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. BabyLM turns 3: Call for papers for the 2025 babylm workshop. Preprint, arXiv:2502… [find](https://scholar.google.com/scholar?q=Lucas+Charpentier%2C+Leshem+Choshen%2C+Ryan+Cotterell%2C+Mustafa+Omer+Gul%2C+Michael+Hu%2C+Jaap+Jumelet%2C+Tal+Linzen%2C+Jing+Liu%2C+Aaron+Mueller%2C+Candace+Ross%2C+Raj+Sanjay+Shah%2C+Alex+Warstadt%2C+Ethan+Wilcox%2C+and+Adin)
4. Jie Chi and Peter Bell. 2024. Analyzing the role of part-of-speech in code-switching: A corpus-based study. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1712–1721, St. Julian’s, Malta. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jie+Chi+and+Peter+Bell%2E+2024%2E+Analyzing+the+role+of+part%2Dof%2Dspeech+in+code%2Dswitching%3A+A+corpus%2Dbased+study%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2024%2C+pages+1712%E2%80%931721%2C+St)
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
6. Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark Dredze. 2022. Bernice: A multilingual pre-trained encoder for Twitter. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6191–6205, Abu Dhabi, United Arab Emirates. Ass… [find](https://scholar.google.com/scholar?q=Alexandra+DeLucia%2C+Shijie+Wu%2C+Aaron+Mueller%2C+Carlos+Aguirre%2C+Philip+Resnik%2C+and+Mark+Dredze%2E+2022%2E+Bernice%3A+A+multilingual+pre%2Dtrained+encoder+for+Twitter%2E+In+Proceedings+of+the+2022+Conference+on+Emp)
7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. [QLORA: efficient finetuning of quantized LLMs](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeuIPS), NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
8. A. Seza Doğruöz, Sunayana Sitaram, and Zheng Xin Yong. 2023. Representativeness as a forgotten lesson for multilingual and code-switched data collection and preparation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5751–5767, Singapore. Association for Computationa… [find](https://scholar.google.com/scholar?q=A%2E+Seza+Do%C4%9Fru%C3%B6z%2C+Sunayana+Sitaram%2C+and+Zheng+Xin+Yong%2E+2023%2E+Representativeness+as+a+forgotten+lesson+for+multilingual+and+code%2Dswitched+data+collection+and+preparation%2E+In+Findings+of+the+Association)
9. Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zi%2DYi+Dou+and+Graham+Neubig%2E+2021%2E+Word+alignment+by+fine%2Dtuning+embeddings+on+parallel+corpora%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Ling)
10. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
11. Markus Frohmann, Igor Sterner, Ivan Vulić, Benjamin Minixhofer, and Markus Schedl. 2024. Segment any Text: A universal approach for robust, efficient and adaptable sentence segmentation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11908–11941, Mia… [find](https://scholar.google.com/scholar?q=Markus+Frohmann%2C+Igor+Sterner%2C+Ivan+Vuli%C4%87%2C+Benjamin+Minixhofer%2C+and+Markus+Schedl%2E+2024%2E+Segment+any+Text%3A+A+universal+approach+for+robust%2C+efficient+and+adaptable+sentence+segmentation%2E+In+Proceeding)
12. Penelope Gardner-Chloros and Daniel Weston. 2015. Code-switching and multilingualism in literature. Language and Literature, 24(3):182–193. [find](https://scholar.google.com/scholar?q=Penelope+Gardner%2DChloros+and+Daniel+Weston%2E+2015%2E+Code%2Dswitching+and+multilingualism+in+literature%2E+Language+and+Literature%2C+24%283%29%3A182%E2%80%93193%2E)
13. François Grosjean. 2010. Bilingual: Life and reality. Harvard university press. [find](https://scholar.google.com/scholar?q=Fran%C3%A7ois+Grosjean%2E+2010%2E+Bilingual%3A+Life+and+reality%2E+Harvard+university+press%2E)
14. John J. Gumperz. 1982. Discourse Strategies. Studies in Interactional Sociolinguistics. Cambridge University Press. [find](https://scholar.google.com/scholar?q=John+J%2E+Gumperz%2E+1982%2E+Discourse+Strategies%2E+Studies+in+Interactional+Sociolinguistics%2E+Cambridge+University+Press%2E)
15. Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In The 2nd BabyLM Challen… [find](https://scholar.google.com/scholar?q=Michael+Y%2E+Hu%2C+Aaron+Mueller%2C+Candace+Ross%2C+Adina+Williams%2C+Tal+Linzen%2C+Chengxu+Zhuang%2C+Ryan+Cotterell%2C+Leshem+Choshen%2C+Alex+Warstadt%2C+and+Ethan+Gotlieb+Wilcox%2E+2024%2E+Findings+of+the+second+BabyLM+cha)
16. Matias Jentoft and David Samuel. 2023. NoCoLA: The Norwegian corpus of linguistic acceptability. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 610–617, Tórshavn, Faroe Islands. University of Tartu Library. [find](https://scholar.google.com/scholar?q=Matias+Jentoft+and+David+Samuel%2E+2023%2E+NoCoLA%3A+The+Norwegian+corpus+of+linguistic+acceptability%2E+In+Proceedings+of+the+24th+Nordic+Conference+on+Computational+Linguistics+%28NoDaLiDa%29%2C+pages+610%E2%80%93617%2C+T%C3%B3)
17. Aravind K. Joshi. 1982. Processing of sentences with intra-sentential code-switching. In Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aravind+K%2E+Joshi%2E+1982%2E+Processing+of+sentences+with+intra%2Dsentential+code%2Dswitching%2E+In+Coling+1982%3A+Proceedings+of+the+Ninth+International+Conference+on+Computational+Linguistics%2E)
18. Jaap Jumelet, Leonie Weissweiler, and Arianna Bisazza. 2025. MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768. [find](https://scholar.google.com/scholar?q=Jaap+Jumelet%2C+Leonie+Weissweiler%2C+and+Arianna+Bisazza%2E+2025%2E+MultiBLiMP+1%2E0%3A+A+massively+multilingual+benchmark+of+linguistic+minimal+pairs%2E+Preprint%2C+arXiv%3A2504%2E02768%2E)
19. Carina Kauf and Anna Ivanova. 2023. [A better way to do masked language model scoring](https://md.research.elamur.ai/free/en/papers/kauf-2023-better-way-do-masked-language/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925–935, Toronto, Canada. Association for Computational Linguistics.
20. Frank Keller and Ash Asudeh. 2002. Probabilistic learning algorithms and optimality theory. Linguistic Inquiry, 33(2):225–244. [find](https://scholar.google.com/scholar?q=Frank+Keller+and+Ash+Asudeh%2E+2002%2E+Probabilistic+learning+algorithms+and+optimality+theory%2E+Linguistic+Inquiry%2C+33%282%29%3A225%E2%80%93244%2E)
21. Prashant Kodali, Anmol Goel, Likhith Asapu, Vamshi Krishna Bonagiri, Anirudh Govil, Monojit Choudhury, Manish Shrivastava, and Ponnurangam Kumaraguru. 2024. [From human judgements to predictive models: Unravelling acceptability in code-mixed sentences](https://md.research.elamur.ai/free/en/papers/kodali-2024-human-judgements-predictive-models-unravelling/). Preprint, arXiv:2405.05572v1.
22. Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. [MADLAD-400: A multilingual and document-level large audited dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Be…
23. Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, and Derry Tanti Wijaya. 2024. Linguistics theory meets LLM: Code-switched text generation via equivalence constrained large language models. Preprint, arXiv:2410.22660v1. [find](https://scholar.google.com/scholar?q=Garry+Kuwanto%2C+Chaitanya+Agarwal%2C+Genta+Indra+Winata%2C+and+Derry+Tanti+Wijaya%2E+2024%2E+Linguistics+theory+meets+LLM%3A+Code%2Dswitched+text+generation+via+equivalence+constrained+large+language+models%2E+Prepr)
24. Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, and Hai Hu. 2024. ZhoBLiMP: a systematic assessment of language models with linguistic minimal pairs in chinese. Preprint, arXiv:2411.06096v1. [find](https://scholar.google.com/scholar?q=Yikang+Liu%2C+Yeting+Shen%2C+Hongao+Zhu%2C+Lilong+Xu%2C+Zhiheng+Qian%2C+Siyuan+Song%2C+Kejia+Zhang%2C+Jialong+Tang%2C+Pei+Zhang%2C+Baosong+Yang%2C+Rui+Wang%2C+and+Hai+Hu%2E+2024%2E+ZhoBLiMP%3A+a+systematic+assessment+of+language)
25. Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xiaojuan Ma, Bertram Shi, and Pascale Fung. 2022. ASCEND: A spontaneous Chinese-English dataset for code-switching in multi-turn conversation. In Proceed… [find](https://scholar.google.com/scholar?q=Holy+Lovenia%2C+Samuel+Cahyawijaya%2C+Genta+Winata%2C+Peng+Xu%2C+Yan+Xu%2C+Zihan+Liu%2C+Rita+Frieske%2C+Tiezheng+Yu%2C+Wenliang+Dai%2C+Elham+J%2E+Barezi%2C+Qifeng+Chen%2C+Xiaojuan+Ma%2C+Bertram+Shi%2C+and+Pascale+Fung%2E+2022%2E+ASC)
26. Dau-Cheng Lyu, Tien-Ping Tan, Eng Siong Chng, and Haizhou Li. 2010. Seame: a mandarin-english code-switching speech corpus in south-east asia. In Eleventh Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Dau%2DCheng+Lyu%2C+Tien%2DPing+Tan%2C+Eng+Siong+Chng%2C+and+Haizhou+Li%2E+2010%2E+Seame%3A+a+mandarin%2Denglish+code%2Dswitching+speech+corpus+in+south%2Deast+asia%2E+In+Eleventh+Annual+Conference+of+the+International+Speech)
27. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. [EuroLLM: Multilingual language m…](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/)
28. MetaAI. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783v3.
29. Pieter Muysken. 1997. Code-switching processes: Alternation, insertion, congruent lexicalization. Language choices: Conditions, constraints, and consequences, 20:361–380. [find](https://scholar.google.com/scholar?q=Pieter+Muysken%2E+1997%2E+Code%2Dswitching+processes%3A+Alternation%2C+insertion%2C+congruent+lexicalization%2E+Language+choices%3A+Conditions%2C+constraints%2C+and+consequences%2C+20%3A361%E2%80%93380%2E)
30. Carol Myers-Scotton. 1993. Social motivations for codeswitching: Evidence from Africa. Oxford University Press. [find](https://scholar.google.com/scholar?q=Carol+Myers%2DScotton%2E+1993%2E+Social+motivations+for+codeswitching%3A+Evidence+from+Africa%2E+Oxford+University+Press%2E)
31. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). In Thirty-seventh Conference on Neural Information Processing Systems.
32. Shana Poplack. 1980. Sometimes I’ll start a sentence in spanish Y TERMINO EN ESPAÑOL: toward a typology of code-switching. Linguistics, 18(7-8):581–618. [find](https://scholar.google.com/scholar?q=Shana+Poplack%2E+1980%2E+Sometimes+I%E2%80%99ll+start+a+sentence+in+spanish+Y+TERMINO+EN+ESPA%C3%91OL%3A+toward+a+typology+of+code%2Dswitching%2E+Linguistics%2C+18%287%2D8%29%3A581%E2%80%93618%2E)
33. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online.…
34. Qwen. 2025. Qwen2.5 technical report. arXiv:2412.15115v2. [find](https://scholar.google.com/scholar?q=Qwen%2E+2025%2E+Qwen2%2E5+technical+report%2E+arXiv%3A2412%2E15115v2%2E)
35. Preprint, Carson T. Schütze. 1996. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press. [find](https://scholar.google.com/scholar?q=Preprint%2C+Carson+T%2E+Sch%C3%BCtze%2E+1996%2E+The+Empirical+Base+of+Linguistics%3A+Grammaticality+Judgments+and+Linguistic+Methodology%2E+University+of+Chicago+Press%2E)
36. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, and Mohit Iyyer. 2022. [SLING: Sino linguistic evaluation of large language models](https://md.research.elamur.ai/free/en/papers/song-2022-sling-sino-linguistic-evaluation-large/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606–4634, Abu Dhabi, United Arab Emirates. Association for Computational Li…
37. Victor Soto, Nishmar Cestero, and Julia Hirschberg. 2018. The role of cognate words, pos tags and entrainment in code-switching. In Interspeech, pages 1938–1942. [find](https://scholar.google.com/scholar?q=Victor+Soto%2C+Nishmar+Cestero%2C+and+Julia+Hirschberg%2E+2018%2E+The+role+of+cognate+words%2C+pos+tags+and+entrainment+in+code%2Dswitching%2E+In+Interspeech%2C+pages+1938%E2%80%931942%2E)
38. Igor Sterner. 2024. Multilingual identification of English code-switching. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 163–173, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Igor+Sterner%2E+2024%2E+Multilingual+identification+of+English+code%2Dswitching%2E+In+Proceedings+of+the+Eleventh+Workshop+on+NLP+for+Similar+Languages%2C+Varieties%2C+and+Dialects+%28VarDial+2024%29%2C+pages+163%E2%80%93173%2C+)
39. Igor Sterner and Simone Teufel. 2023. TongueSwitcher: Fine-grained identification of German-English code-switching. In Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching, pages 1–13, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Igor+Sterner+and+Simone+Teufel%2E+2023%2E+TongueSwitcher%3A+Fine%2Dgrained+identification+of+German%2DEnglish+code%2Dswitching%2E+In+Proceedings+of+the+6th+Workshop+on+Computational+Approaches+to+Linguistic+Code%2DSw)
40. Igor Sterner and Simone Teufel. 2025. Code-switching and syntax: A large–scale experiment. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Igor+Sterner+and+Simone+Teufel%2E+2025%2E+Code%2Dswitching+and+syntax%3A+A+large%E2%80%93scale+experiment%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2025%2C+Vienna%2C+Austria%2E+Association+for+Comp)
41. Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L. Frank. 2025. BLiMP-NL: A corpus of dutch minimal pairs and acceptability judgments for language model evaluation. Computational Linguistics, pages 1–35. [find](https://scholar.google.com/scholar?q=Michelle+Suijkerbuijk%2C+Zo%C3%AB+Prins%2C+Marianne+de+Heer+Kloots%2C+Willem+Zuidema%2C+and+Stefan+L%2E+Frank%2E+2025%2E+BLiMP%2DNL%3A+A+corpus+of+dutch+minimal+pairs+and+acceptability+judgments+for+language+model+evaluatio)
42. Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, Ekaterina Artemova, and Vladislav Mikhailov. 2024. RuBLiMP: Russian benchmark of linguistic minimal pairs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9268–9299, Miami, Florida… [find](https://scholar.google.com/scholar?q=Ekaterina+Taktasheva%2C+Maxim+Bazhukov%2C+Kirill+Koncha%2C+Alena+Fenogenova%2C+Ekaterina+Artemova%2C+and+Vladislav+Mikhailov%2E+2024%2E+RuBLiMP%3A+Russian+benchmark+of+linguistic+minimal+pairs%2E+In+Proceedings+of+the+)
43. Naoya Ueda, Masato Mita, Teruaki Oka, and Mamoru Komachi. 2024. Token-length bias in minimal-pair paradigm datasets. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16224–16236, Torino, Italia. ELRA a… [find](https://scholar.google.com/scholar?q=Naoya+Ueda%2C+Masato+Mita%2C+Teruaki+Oka%2C+and+Mamoru+Komachi%2E+2024%2E+Token%2Dlength+bias+in+minimal%2Dpair+paradigm+datasets%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computational+Linguist)
44. Elena Volodina, Yousuf Ali Mohammed, and Julia Klezl. 2021. DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing. Preprint, arXiv:2105.06681v1. [find](https://scholar.google.com/scholar?q=Elena+Volodina%2C+Yousuf+Ali+Mohammed%2C+and+Julia+Klezl%2E+2021%2E+DaLAJ+%2D+a+dataset+for+linguistic+acceptability+judgments+for+Swedish%3A+Format%2C+baseline%2C+sharing%2E+Preprint%2C+arXiv%3A2105%2E06681v1%2E)
45. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mosquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+and+Ryan+Cotterell%2E+2023%2E+Findings+of+the+BabyLM)
46. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. [BLiMP: The benchmark of linguistic minimal pairs for English](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377–392.
47. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. [CLiMP: A benchmark for Chinese language model evaluation](https://md.research.elamur.ai/free/en/papers/xiang-2021-climp-benchmark-chinese-language-model/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo…
48. Tatu Ylonen. 2022. Wiktextract: Wiktionary as machine-readable structured data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1317–1325. [find](https://scholar.google.com/scholar?q=Tatu+Ylonen%2E+2022%2E+Wiktextract%3A+Wiktionary+as+machine%2Dreadable+structured+data%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluation+Conference%2C+pages+1317%E2%80%931325%2E)
49. Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Aji. 2023. [Prompting multilingual large language mode…](https://md.research.elamur.ai/free/en/papers/yong-2023-prompting-multilingual-large-language-models/)
50. Özlem Çetinoğlu. 2016. A Turkish-German code-switching corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 23–28, Paris, France. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=%C3%96zlem+%C3%87etino%C4%9Flu%2E+2016%2E+A+Turkish%2DGerman+code%2Dswitching+corpus%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2016%29%2C+pages+23%E2%80%9328%2C+Paris%2C+France%2E+Europe)

*Updated 31 October 2026*