---
title: "Estonian WinoGrande Dataset — Ojastu 2026"
description: "WinoGrande is a common benchmark for commonsense reasoning in LLMs, but it exists only in English."
authors: ["Marii Ojastu", "Hele-Andra Kuulmets", "Aleksei Dorkin", "Marika Borovikova", "Dage Särg", "Kairit Sirts"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.549/"
doi: "10.63317/27gw7fzp2jbx"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ojastu-2026-estonian-winogrande-dataset-comparative-analysis/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ojastu-2026-estonian-winogrande-dataset-comparative-analysis/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/2/) / Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation

# Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation

**Authors:** Marii Ojastu, Hele-Andra Kuulmets, Aleksei Dorkin, Marika Borovikova, Dage Särg, [Kairit Sirts](https://md.research.elamur.ai/free/authors/kairit-sirts/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.549/) · [PDF](https://aclanthology.org/2026.lrec-1.549.pdf) · [DOI 10.63317/27gw7fzp2jbx](https://doi.org/10.63317/27gw7fzp2jbx) · [OpenAlex](https://openalex.org/W4416597228)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · 3 in our library, of the 140673 papers whose bibliographies we read

## Summary
WinoGrande is a common benchmark for commonsense reasoning in LLMs, but it exists only in English. Benchmarks for other languages are often produced by machine translation, which can distort meaning and cannot localize content. The authors had translation specialists create an Estonian version of the WinoGrande test set (1,767 items). The specialists kept the benchmark's structural constraints, such as lexical overlap between twin sentences and identical answer options. They also adapted culturally specific items and fixed flawed or ambiguous originals.

They then built two machine-translated versions with OpenAI models, one from a simple prompt and one from a detailed prompt based on lessons from the manual work. They evaluated proprietary and open models on all versions.

Results:
- Proprietary models score about the same on English and human-translated Estonian.
- Open models do slightly worse in Estonian.
- Every model does clearly worse on the machine-translated data.
- The detailed prompt helps only a little.

The study shows that language experts are still needed to build trustworthy non-English reasoning benchmarks.

## Key points
- The human-translated Estonian WinoGrande test set is released at huggingface.co/datasets/tartuNLP/winogrande_et. Of its 1,767 schemas, 89 (5%) were corrected and 53 (3%) were localized or culturally adapted.
- Manual checking found that machine translation changed or lost the meaning of 246 schemas (15.2%) with the simple prompt and 191 (11.8%) with the detailed prompt.
- On schemas whose meaning shifted, average accuracy fell by 4.7% (simple prompt) and 7.2% (detailed prompt). Qwen 2.5-72B was the only model that did not get worse.
- On schemas whose meaning was kept, models still scored 3.8% and 3.3% lower on the machine-translated text than on the human translation. This suggests other translation artifacts also hurt task integrity.
- Most smaller models performed poorly. The exception was EuroLLM-9B, which is trained specifically for European languages. Annotator accuracy stayed close to the 94% human level reported for the English test set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ojastu2026estonian,
  title = {{Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation}},
  author = {Ojastu, Marii and Kuulmets, Hele-Andra and Dorkin, Aleksei and Borovikova, Marika and Särg, Dage and Sirts, Kairit},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/27gw7fzp2jbx},
  url = {https://doi.org/10.63317/27gw7fzp2jbx}
}
```

- [← previous Ensemble Methods for Low-Resource Russian-Kyrgyz Machine Translation: When Diverse Models Beat Better Models](https://md.research.elamur.ai/free/papers/metinov-2026-ensemble-methods-low-resource-russian/)
- [next → Evaluating Discriminability of Vision-Language Models](https://md.research.elamur.ai/free/papers/muraoka-2026-evaluating-discriminability-vision-language-models/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/2/)

## References

57 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tuka Alhanai, Adam Kasumovic, Mohammad M. Ghassemi, Aven Zitzelberger, Jessica M. Lundin, and Guillaume Chabot-Couture. 2025. Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments. Proceedings of the AAAI Conference… [find](https://scholar.google.com/scholar?q=Tuka+Alhanai%2C+Adam+Kasumovic%2C+Mohammad+M%2E+Ghassemi%2C+Aven+Zitzelberger%2C+Jessica+M%2E+Lundin%2C+and+Guillaume+Chabot%2DCouture%2E+2025%2E+Bridging+the+Gap%3A+Enhancing+LLM+Performance+for+Low%2DResource+African+Langu)
2. Pascal Amsili and Olga Seminck. 2017. A Google-Proof Collection of French Winograd Schemas. [find](https://scholar.google.com/scholar?q=Pascal+Amsili+and+Olga+Seminck%2E+2017%2E+A+Google%2DProof+Collection+of+French+Winograd+Schemas%2E)
3. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 24–29, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2nd+Workshop+on+Coreference+Resolution+Beyond+OntoNotes+%28CORBON+2017%29%2C+pages+24%E2%80%9329%2C+Valencia%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
4. Anthropic. 2025. System Card: Claude Sonnet 4.5. [https://assets.anthropic.com/m/12f214efcc2f457a/original/](https://assets.anthropic.com/m/12f214efcc2f457a/original/) Claude-Sonnet-4-5-System-Card.pdf. [↗](https://assets.anthropic.com/m/12f214efcc2f457a/original/)
5. Maksim Aparovich, Volha Harytskaya, Vladislav Poritski, Oksana Volchek, and Pavel Smrz. 2025. BelarusianGLUE: Towards a Natural Language Understanding Benchmark for Belarusian. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5… [find](https://scholar.google.com/scholar?q=Maksim+Aparovich%2C+Volha+Harytskaya%2C+Vladislav+Poritski%2C+Oksana+Volchek%2C+and+Pavel+Smrz%2E+2025%2E+BelarusianGLUE%3A+Towards+a+Natural+Language+Understanding+Benchmark+for+Belarusian%2E+In+Proceedings+of+the+6)
6. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation Artifacts in Cross-lingual Transfer Learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674–7684, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+and+Eneko+Agirre%2E+2020%2E+Translation+Artifacts+in+Cross%2Dlingual+Transfer+Learning%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
7. Phakphum Artkaew. 2025. Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning. In Proceedings of the Second Workshop in South East Asian Language Processing, pages 42–51, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Phakphum+Artkaew%2E+2025%2E+Thai+Winograd+Schemas%3A+A+Benchmark+for+Thai+Commonsense+Reasoning%2E+In+Proceedings+of+the+Second+Workshop+in+South+East+Asian+Language+Processing%2C+pages+42%E2%80%9351%2C+Online%2E+Associati)
8. Timothée Bernard and Ting Han. 2020. Mandarinograd: A Chinese Collection of Winograd Schemas. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 21–26, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Timoth%C3%A9e+Bernard+and+Ting+Han%2E+2020%2E+Mandarinograd%3A+A+Chinese+Collection+of+Winograd+Schemas%2E+In+Proceedings+of+the+Twelfth+Language+Resources+and+Evaluation+Conference%2C+pages+21%E2%80%9326%2C+Marseille%2C+France)
9. Ander Corral, Ixak Sarasua, and Xabier Saralegi. 2024. Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque. [find](https://scholar.google.com/scholar?q=Ander+Corral%2C+Ixak+Sarasua%2C+and+Xabier+Saralegi%2E+2024%2E+Pipeline+Analysis+for+Developing+Instruct+LLMs+in+Low%2DResource+Languages%3A+A+Case+Study+on+Basque%2E)
10. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261. [find](https://scholar.google.com/scholar?q=John+Dang%2C+Shivalika+Singh%2C+Daniel+D%E2%80%99souza%2C+Arash+Ahmadian%2C+Alejandro+Salamanca%2C+Madeline+Smith%2C+Aidan+Peppin%2C+Sungjin+Hong%2C+Manoj+Govindassamy%2C+Terrence+Zhao%2C+et+al%2E+2024%2E+Aya+expanse%3A+Combining+rese) [our card](https://md.research.elamur.ai/free/papers/dang-2024-aya-expanse-combining-research-breakthroughs/)
11. Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021. Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10486–10500, Online and Punta Can… [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Hongming+Zhang%2C+Yoav+Goldberg%2C+and+Dan+Roth%2E+2021%2E+Back+to+Square+One%3A+Artifact+Detection%2C+Training+and+Commonsense+Disentanglement+in+the+Winograd+Schema%2E+In+Proceedings+of+the+2021+Con)
12. Farhan Farsi, Farnaz Aghababaloo, Shahriar Shariati Motlagh, Parsa Ghofrani, MohammadAli SadraeiJavaheri, Shayan Bali, Amirhossein Shabani, Farbod Bijary, Ghazal Zamaninejad, AmirMohammad Salehoof, and Saeedeh Momtazi. 2025. MELAC: Massive Evaluation of Large Language Models with Alignment of Cultur… [find](https://scholar.google.com/scholar?q=Farhan+Farsi%2C+Farnaz+Aghababaloo%2C+Shahriar+Shariati+Motlagh%2C+Parsa+Ghofrani%2C+MohammadAli+SadraeiJavaheri%2C+Shayan+Bali%2C+Amirhossein+Shabani%2C+Farbod+Bijary%2C+Ghazal+Zamaninejad%2C+AmirMohammad+Salehoof%2C+an)
13. Negar Foroutan, Mohammadreza Banaei, Karl Aberer, and Antoine Bosselut. 2023. Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9422–9442, Singapore. Association for Computa… [find](https://scholar.google.com/scholar?q=Negar+Foroutan%2C+Mohammadreza+Banaei%2C+Karl+Aberer%2C+and+Antoine+Bosselut%2E+2023%2E+Breaking+the+Language+Barrier%3A+Improving+Cross%2DLingual+Reasoning+with+Structured+Self%2DAttention%2E+In+Findings+of+the+Associ)
14. Gemini Team, Google. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261. [find](https://scholar.google.com/scholar?q=Gemini+Team%2C+Google%2E+2025%2E+Gemini+2%2E5%3A+Pushing+the+Frontier+with+Advanced+Reasoning%2C+Multimodality%2C+Long+Context%2C+and+Next+Generation+Agentic+Capabilities%2E+arXiv+preprint+arXiv%3A2507%2E06261%2E) [our card](https://md.research.elamur.ai/free/papers/comanici-2025-gemini-2-5-pushing-frontier/)
15. Google Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. [find](https://scholar.google.com/scholar?q=Google+Gemma+Team%2E+2024%2E+Gemma+2%3A+Improving+open+language+models+at+a+practical+size%2E+arXiv+preprint+arXiv%3A2408%2E00118%2E)
16. Google DeepMind Gemma Gemma 3 technical report. arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Google+DeepMind+Gemma+Gemma+3+technical+report%2E+arXiv%3A2503%2E19786%2E)
17. Team. 2025. arXiv preprint Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. [find](https://scholar.google.com/scholar?q=Team%2E+2025%2E+arXiv+preprint+Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Alex+Vaughan%2C+et+al%2E+2024%2E+T)
18. Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Mic… [find](https://scholar.google.com/scholar?q=Alejandro+Hern%C3%A1ndez%2DCano%2C+Alexander+H%C3%A4gele%2C+Allen+Hao+Huang%2C+Angelika+Romanou%2C+Antoni%2DJoan+Solergibert%2C+Barna+Pasztor%2C+Bettina+Messmer%2C+Dhia+Garbaya%2C+Eduard+Frank+%C4%8Eurech%2C+Ido+Hakimi%2C+Juan+Garc%C3%ADa+Giral)
19. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of Experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
20. Hyeonwoo Kim, Dahyun Kim, Jihoo Kim, Sukyung Lee, Yungi Kim, and Chanjun Park. 2025. Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs. [find](https://scholar.google.com/scholar?q=Hyeonwoo+Kim%2C+Dahyun+Kim%2C+Jihoo+Kim%2C+Sukyung+Lee%2C+Yungi+Kim%2C+and+Chanjun+Park%2E+2025%2E+Open+Ko%2DLLM+Leaderboard2%3A+Bridging+Foundational+and+Practical+Evaluation+for+Korean+LLMs%2E)
21. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A Surprisingly Robust Trick for the Winograd Schema Challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837–4842, Florence, Italy. Association fo… [find](https://scholar.google.com/scholar?q=Vid+Kocijan%2C+Ana%2DMaria+Cretu%2C+Oana%2DMaria+Camburu%2C+Yordan+Yordanov%2C+and+Thomas+Lukasiewicz%2E+2019%2E+A+Surprisingly+Robust+Trick+for+the+Winograd+Schema+Challenge%2E+In+Proceedings+of+the+57th+Annual+Meetin)
22. Vid Kocijan, Ernest Davis, Thomas Lukasiewicz, Gary Marcus, and Leora Morgenstern. 2023. The Defeat of the Winograd Schema Challenge. [find](https://scholar.google.com/scholar?q=Vid+Kocijan%2C+Ernest+Davis%2C+Thomas+Lukasiewicz%2C+Gary+Marcus%2C+and+Leora+Morgenstern%2E+2023%2E+The+Defeat+of+the+Winograd+Schema+Challenge%2E)
23. Julia Kreutzer, Eleftheria Briakou, Sweta Agrawal, Marzieh Fadaee, and Kocmi Tom. 2025. D\\’ej\\a vu: Multilingual llm evaluation through the lens of machine translation evaluation. arXiv preprint arXiv:2504.11829. [find](https://scholar.google.com/scholar?q=Julia+Kreutzer%2C+Eleftheria+Briakou%2C+Sweta+Agrawal%2C+Marzieh+Fadaee%2C+and+Kocmi+Tom%2E+2025%2E+D%5C%5C%E2%80%99ej%5C%5Ca+vu%3A+Multilingual+llm+evaluation+through+the+lens+of+machine+translation+evaluation%2E+arXiv+preprint+arX)
24. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback. [find](https://scholar.google.com/scholar?q=Viet+Lai%2C+Chien+Nguyen%2C+Nghia+Ngo%2C+Thuat+Nguyen%2C+Franck+Dernoncourt%2C+Ryan+Rossi%2C+and+Thien+Nguyen%2E+2023%2E+Okapi%3A+Instruction%2Dtuned+Large+Language+Models+in+Multiple+Languages+with+Reinforcement+Learnin) [our card](https://md.research.elamur.ai/free/papers/lai-2023-okapi-instruction-tuned-large-language/)
25. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%3A+System+Demonstrations%2C+pages+318%E2%80%93327%2C+Singapore%2E+Association+for+Computational+Linguistics%2E)
26. J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, pages 159–174. [find](https://scholar.google.com/scholar?q=J+Richard+Landis+and+Gary+G+Koch%2E+1977%2E+The+Measurement+of+Observer+Agreement+for+Categorical+Data%2E+Biometrics%2C+pages+159%E2%80%93174%2E)
27. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, page 552–561. AAAI Press. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+Schema+Challenge%2E+In+Proceedings+of+the+Thirteenth+International+Conference+on+Principles+of+Knowledge+Representation+and+Re)
28. AI @ Meta Llama Team. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. [find](https://scholar.google.com/scholar?q=AI+%40+Meta+Llama+Team%2E+2024%2E+The+Llama+3+Herd+of+Models%2E+arXiv+preprint+arXiv%3A2407%2E21783%2E)
29. Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13480–13488. [find](https://scholar.google.com/scholar?q=Nicholas+Lourie%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+UNICORN+on+RAINBOW%3A+A+Universal+Commonsense+Reasoning+Model+on+a+New+Multitask+Benchmark%2E+Proceedings+of+the+AAAI+Conference+)
30. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. EuroLLM: Multilingual Language M… [find](https://scholar.google.com/scholar?q=Pedro+Henrique+Martins%2C+Patrick+Fernandes%2C+Jo%C3%A3o+Alves%2C+Nuno+M%2E+Guerreiro%2C+Ricardo+Rei%2C+Duarte+M%2E+Alves%2C+Jos%C3%A9+Pombal%2C+Amin+Farajian%2C+Manuel+Faysse%2C+Mateusz+Klimaszewski%2C+Pierre+Colombo%2C+Barry+Haddow%2C+J) [our card](https://md.research.elamur.ai/free/papers/martins-2024-eurollm-multilingual-language-models-europe/)
31. Mihai Masala, Denis Ilie-Ablachim, Alexandru Dima, Dragos Georgian Corlatescu, Miruna-Andreea Zavelca, Ovio Olaru, Simina-Maria Terian, Andrei Terian, Marius Leordeanu, Horia Velicu, Marius Popescu, Mihai Dascalu, and Traian Rebedea. 2024. “Vorbes , ti Românes , te?” A Recipe to Train Powerful Roman… [find](https://scholar.google.com/scholar?q=Mihai+Masala%2C+Denis+Ilie%2DAblachim%2C+Alexandru+Dima%2C+Dragos+Georgian+Corlatescu%2C+Miruna%2DAndreea+Zavelca%2C+Ovio+Olaru%2C+Simina%2DMaria+Terian%2C+Andrei+Terian%2C+Marius+Leordeanu%2C+Horia+Velicu%2C+Marius+Popescu%2C+M)
32. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+2019%2E+Right+for+the+Wrong+Reasons%3A+Diagnosing+Syntactic+Heuristics+in+Natural+Language+Inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associ) [our card](https://md.research.elamur.ai/free/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
33. Gabriela Melo, Vinicius Imaizumi, and Fábio Cozman. 2019. Winograd Schemas in Portuguese. In Anais do XVI Encontro Nacional de Inteligência Artificial e Computacional, pages 787–798, Porto Alegre, RS, Brasil. SBC. [find](https://scholar.google.com/scholar?q=Gabriela+Melo%2C+Vinicius+Imaizumi%2C+and+F%C3%A1bio+Cozman%2E+2019%2E+Winograd+Schemas+in+Portuguese%2E+In+Anais+do+XVI+Encontro+Nacional+de+Intelig%C3%AAncia+Artificial+e+Computacional%2C+pages+787%E2%80%93798%2C+Porto+Alegre%2C+RS%2C)
34. Luca Moroni, Simone Conia, Federico Martelli, and Roberto Navigli. 2024. Towards a More Comprehensive Evaluation for Italian LLMs. In Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), pages 584–599, Pisa, Italy. CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Luca+Moroni%2C+Simone+Conia%2C+Federico+Martelli%2C+and+Roberto+Navigli%2E+2024%2E+Towards+a+More+Comprehensive+Evaluation+for+Italian+LLMs%2E+In+Proceedings+of+the+Tenth+Italian+Conference+on+Computational+Lingu)
35. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics,… [find](https://scholar.google.com/scholar?q=Basel+Mousi%2C+Nadir+Durrani%2C+Fatema+Ahmad%2C+Md%2E+Arid+Hasan%2C+Maram+Hasanain%2C+Tameem+Kabbani%2C+Fahim+Dalvi%2C+Shammur+Absar+Chowdhury%2C+and+Firoj+Alam%2E+2025%2E+AraDiCE%3A+Benchmarks+for+dialectal+and+cultural+cap)
36. Artūras Nakvosas, Povilas Daniušis, and Vytas Mulevičius. 2025. Open Llama2 Models for the Lithuanian Language. Informatica, page 385–406. [find](https://scholar.google.com/scholar?q=Artu%CC%84ras+Nakvosas%2C+Povilas+Daniu%C5%A1is%2C+and+Vytas+Mulevi%C4%8Dius%2E+2025%2E+Open+Llama2+Models+for+the+Lithuanian+Language%2E+Informatica%2C+page+385%E2%80%93406%2E)
37. Jinu Nyachhyon, Mridul Sharma, Prajwal Thapa, and Bal Krishna Bal. 2025. Consolidating and Developing Benchmarking Datasets for the Nepali Natural Language Understanding Tasks. [find](https://scholar.google.com/scholar?q=Jinu+Nyachhyon%2C+Mridul+Sharma%2C+Prajwal+Thapa%2C+and+Bal+Krishna+Bal%2E+2025%2E+Consolidating+and+Developing+Benchmarking+Datasets+for+the+Nepali+Natural+Language+Understanding+Tasks%2E)
38. OpenAI. 2025. GPT-5 System Card. [https://cdn.openai.com/gpt-5-system-card](https://cdn.openai.com/gpt-5-system-card). pdf. [↗](https://cdn.openai.com/gpt-5-system-card)
39. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… [find](https://scholar.google.com/scholar?q=OpenAI%2C+Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+Red+Avila%2C+Igor+Babuschkin%2C+Suchir) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
40. Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor-Rocher, and María Grandury. 2024. Spanish and LLM Benchmarks: is MMLU Lost in Translation? [find](https://scholar.google.com/scholar?q=Irene+Plaza%2C+Nina+Melero%2C+Cristina+del+Pozo%2C+Javier+Conde%2C+Pedro+Reviriego%2C+Marina+Mayor%2DRocher%2C+and+Mar%C3%ADa+Grandury%2E+2024%2E+Spanish+and+LLM+Benchmarks%3A+is+MMLU+Lost+in+Translation%3F)
41. Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2025. mHumanEval - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Languag… [find](https://scholar.google.com/scholar?q=Nishat+Raihan%2C+Antonios+Anastasopoulos%2C+and+Marcos+Zampieri%2E+2025%2E+mHumanEval+%2D+A+Multilingual+Benchmark+to+Evaluate+Large+Language+Models+for+Code+Generation%2E+In+Proceedings+of+the+2025+Conference+of)
42. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+2018%2E+Gender+Bias+in+Coreference+Resolution%2E) [our card](https://md.research.elamur.ai/free/papers/rudinger-2018-gender-bias-coreference-resolution/)
43. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2019%2E+WinoGrande%3A+An+Adversarial+Winograd+Schema+Challenge+at+Scale%2E)
44. Guokan Shang, Hadi Abdine, Ahmad Chamma, Amr Mohamed, Mohamed Anwar, Abdelaziz Bounhar, Omar El Herraoui, Preslav Nakov, Michalis Vazirgiannis, and Eric Xing. 2025. Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts. [find](https://scholar.google.com/scholar?q=Guokan+Shang%2C+Hadi+Abdine%2C+Ahmad+Chamma%2C+Amr+Mohamed%2C+Mohamed+Anwar%2C+Abdelaziz+Bounhar%2C+Omar+El+Herraoui%2C+Preslav+Nakov%2C+Michalis+Vazirgiannis%2C+and+Eric+Xing%2E+2025%2E+Nile%2DChat%3A+Egyptian+Language+Models)
45. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark. In Proceedings of the 2020 Conference o… [find](https://scholar.google.com/scholar?q=Tatiana+Shavrina%2C+Alena+Fenogenova%2C+Emelyanov+Anton%2C+Denis+Shevelev%2C+Ekaterina+Artemova%2C+Valentin+Malykh%2C+Vladislav+Mikhailov%2C+Maria+Tikhonova%2C+Andrey+Chertok%2C+and+Andrey+Evlampiev%2E+2020%2E+RussianSuper)
46. Tomohide Shibata, Shotaro Kohama, and Sadao Kurohashi. 2015. winograd schema challenge. In Proceedings of the Annual Meeting of the Association for Natural Language Processing (NLP2015), pages 493–496, Kyoto, Japan. Association for Natural Language Processing. [find](https://scholar.google.com/scholar?q=Tomohide+Shibata%2C+Shotaro+Kohama%2C+and+Sadao+Kurohashi%2E+2015%2E+winograd+schema+challenge%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Natural+Language+Processing+%28NLP2015%29%2C+pages+493%E2%80%93496)
47. Vered Shwartz. 2024. Winograd schema challenge datasets. [https://www.cs.ubc.ca/~vshwartz/resources/winograd_he](https://www.cs.ubc.ca/~vshwartz/resources/winograd_he). html. Accessed: 2026-02-27. [↗](https://www.cs.ubc.ca/~vshwartz/resources/winograd_he)
48. Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma O… [find](https://scholar.google.com/scholar?q=Shivalika+Singh%2C+Freddie+Vargus%2C+Daniel+D%E2%80%99souza%2C+B%C3%B6rje+F%2E+Karlsson%2C+Abinaya+Mahendiran%2C+Wei%2DYin+Ko%2C+Herumb+Shandilya%2C+Jay+Patel%2C+Deividas+Mataciunas%2C+Laura+O%E2%80%99Mahony%2C+Mike+Zhang%2C+Ramith+Hettiarachchi%2C+) [our card](https://md.research.elamur.ai/free/papers/singh-2024-aya-dataset-open-access-collection/)
49. Vésteinn Snæbjarnarson, Haukur Barri Símonarson, Pétur Orri Ragnarsson, Svanhvít Lilja Ingólfsdóttir, Haukur Jónsson, Vilhjalmur Thorsteinsson, and Hafsteinn Einarsson. 2022. A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models. In Proceedings of the Thirteenth Language Resour… [find](https://scholar.google.com/scholar?q=V%C3%A9steinn+Sn%C3%A6bjarnarson%2C+Haukur+Barri+S%C3%ADmonarson%2C+P%C3%A9tur+Orri+Ragnarsson%2C+Svanhv%C3%ADt+Lilja+Ing%C3%B3lfsd%C3%B3ttir%2C+Haukur+J%C3%B3nsson%2C+Vilhjalmur+Thorsteinsson%2C+and+Hafsteinn+Einarsson%2E+2022%2E+A+Warm+Start+and+a+Clean+)
50. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. 2024. Towards Multilingual LLM Evaluation for European Languages. [find](https://scholar.google.com/scholar?q=Klaudia+Thellmann%2C+Bernhard+Stadler%2C+Michael+Fromm%2C+Jasper+Schulze+Buschhoff%2C+Alex+Jude%2C+Fabio+Barth%2C+Johannes+Leveling%2C+Nicolas+Flores%2DHerr%2C+Joachim+K%C3%B6hler%2C+Ren%C3%A9+J%C3%A4kel%2C+and+Mehdi+Ali%2E+2024%2E+Towards+M)
51. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+Foundation+and+Fi)
52. Timur Turatali, Aida Turdubaeva, Islam Zhenish-bekov, Zhoomart Suranbaev, Anton Alekseev, and Rustem Izmailov. 2025. Bridging the Gap in Less-Resourced Languages: Building a Benchmark for Kyrgyz Language Models. In 2025 10th International Conference on Computer Science and Engineering (UBMK), pages… [find](https://scholar.google.com/scholar?q=Timur+Turatali%2C+Aida+Turdubaeva%2C+Islam+Zhenish%2Dbekov%2C+Zhoomart+Suranbaev%2C+Anton+Alekseev%2C+and+Rustem+Izmailov%2E+2025%2E+Bridging+the+Gap+in+Less%2DResourced+Languages%3A+Building+a+Benchmark+for+Kyrgyz+Langu)
53. Noémi Vadász and Noémi Ligeti-Nagy. 2022. Winograd Schemata and Other Datasets for Anaphora Resolution in Hungarian. Acta Linguistica Academica. [find](https://scholar.google.com/scholar?q=No%C3%A9mi+Vad%C3%A1sz+and+No%C3%A9mi+Ligeti%2DNagy%2E+2022%2E+Winograd+Schemata+and+Other+Datasets+for+Anaphora+Resolution+in+Hungarian%2E+Acta+Linguistica+Academica%2E)
54. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+SuperGLUE%3A+A+Stickier+Benchmark+for+General%2DPurpose+Language+Understan) [our card](https://md.research.elamur.ai/free/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
55. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Anfeng+Li%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Gao%2C+Chengen+Huang%2C+Chenxu+Lv%2C+Chujie+Zheng%2C+Dayiheng+Liu%2C+Fan+Zhou%2C+Fei+Huang%2C+Feng+Hu%2C+Hao+Ge%2C+Haoran+Wei%2C+Hua) [our card](https://md.research.elamur.ai/free/papers/yang-2025-qwen3-technical-report/)
56. Aleš Žagar and Marko Robnik-Šikonja. 2022. Slovene SuperGLUE benchmark: Translation and evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2058–2065, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Ale%C5%A1+%C5%BDagar+and+Marko+Robnik%2D%C5%A0ikonja%2E+2022%2E+Slovene+SuperGLUE+benchmark%3A+Translation+and+evaluation%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluation+Conference%2C+pages+2058%E2%80%932065%2C+Mars)
57. Hongming Zhang, Xinran Zhao, and Yangqiu Song. 2020. WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5736–5745, Online. Association for Computationa… [find](https://scholar.google.com/scholar?q=Hongming+Zhang%2C+Xinran+Zhao%2C+and+Yangqiu+Song%2E+2020%2E+WinoWhy%3A+A+Deep+Diagnosis+of+Essential+Commonsense+Knowledge+for+Answering+Winograd+Schema+Challenge%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+)