---
title: "Serbian SuperGLUE — paper by Perovic and Mihajlov (2026)"
description: "Serbian, and BCMS languages more broadly, have no standard way to test language models on… 52 references, cited by 1 paper here."
authors: ["Mitar Perovic", "Teodora Mihajlov"]
year: 2026
venue: "Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)"
source: "https://aclanthology.org/2026.loreslm-1.30/"
doi: "10.18653/v1/2026.loreslm-1.30"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/perovic-2026-serbian-superglue-evaluation-benchmark-south/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/perovic-2026-serbian-superglue-evaluation-benchmark-south/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/) / Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models

# Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models

**Authors:** Mitar Perovic and Teodora Mihajlov · **Year:** 2026 · **Venue:** [Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)](https://md.research.elamur.ai/free/en/venues/loreslm/) ([Workshop on Language Models for Low-Resource Languages 2026](https://md.research.elamur.ai/free/en/venues/loreslm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.loreslm-1.30/) · [PDF](https://aclanthology.org/2026.loreslm-1.30.pdf) · [DOI 10.18653/v1/2026.loreslm-1.30](https://doi.org/10.18653/v1/2026.loreslm-1.30) · [OpenAlex](https://openalex.org/W7140129886)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Serbian, and BCMS languages more broadly, have no standard way to test language models on demanding understanding tasks. The only regional suite, BENCHić, covers simpler problems such as NER and sentiment. The authors fill this gap by adapting the English SuperGLUE into Serbian. They chose Gemini Flash 2.5 as the translator after annotators compared five LLMs, then applied automatic post-editing. Some tasks needed other routes: COPA was taken from BENCHić, ReCoRD was built from existing resources, and WiC was left out because no Serbian lexicon is available. Diagnostic subsets were checked by hand.

The authors fine-tuned seven encoder models on the resulting seven-task suite, comparing Serbian-specific and multilingual models. Models built only for Serbian did not consistently beat multilingual ones. Results on small datasets swung widely between random seeds, so the authors argue that multi-seed reporting is essential in low-resource evaluation. The benchmark, code and checkpoints are released openly as a shared base for future South Slavic NLU work.

## Key points
- The suite has seven tasks: BoolQ, CB, COPA, MultiRC, ReCoRD, RTE and WSC. They span QA, NLI and coreference, and WiC is excluded.
- Seven encoders were evaluated, including BERTić, Jerteh, mmBERT, XLM-R-BERTić and TeslaXLM.
  - mmBERT scored best on RTE at 75.7%, and XLM-R-BERTić led on BoolQ at 82.0%.
  - BERTić was strong on MultiRC (F1a 65.8%) and on BoolQ (77.1%).
  - Jerteh (355M) did well on CB but failed on MultiRC.
- Results varied sharply across seeds. XLM-R-BERTić on RTE ranged from 52.7% to 79.8% (σ = 12.7%). The authors recommend reporting results over at least 5 seeds and using a learning rate of 5×10⁻⁶ for datasets under 1,000 examples.
- ReCoRD proved easiest, with all models at 94–96%, which likely reflects simpler generated questions.
- WSC did not separate the models, which all scored about 63–65%. The authors cite three causes:
  - only 554 training examples;
  - roughly 65% positive labels;
  - translation artifacts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Mitar Perovic and Teodora Mihajlov. 2026. Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{perovic2026serbian,
  title = {{Serbian SuperGLUE: Towards an Evaluation Benchmark for South Slavic Language Models}},
  author = {Mitar Perovic and Teodora Mihajlov},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.30},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.30}
}
```

- [← previous SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs](https://md.research.elamur.ai/free/en/papers/sun-2026-sepseq-training-free-framework-long/)
- [next → SharVeT: Similarity-aware Parameter Sharing with Vector-based Tuning for Efficient LLM Compression](https://md.research.elamur.ai/free/en/papers/yun-2026-sharvet-similarity-aware-parameter-sharing/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/)

## References

52 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yvonne Adesam, Aleksandrs Berdicevskis, and Felix Morger. 2020. Swedishglue–towards a swedish test set for evaluating natural language understanding models. [find](https://scholar.google.com/scholar?q=Yvonne+Adesam%2C+Aleksandrs+Berdicevskis%2C+and+Felix+Morger%2E+2020%2E+Swedishglue%E2%80%93towards+a+swedish+test+set+for+evaluating+natural+language+understanding+models%2E)
2. Maksim Aparovich, Volha Harytskaya, Vladislav Poritski, Oksana Volchek, and Pavel Smrz. 2025. Belarusianglue: Towards a natural language understanding benchmark for belarusian. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5… [find](https://scholar.google.com/scholar?q=Maksim+Aparovich%2C+Volha+Harytskaya%2C+Vladislav+Poritski%2C+Oksana+Volchek%2C+and+Pavel+Smrz%2E+2025%2E+Belarusianglue%3A+Towards+a+natural+language+understanding+benchmark+for+belarusian%2E+In+Proceedings+of+the+6)
3. Marta Bañón, Miquel Esplà-Gomis, Mikel L Forcada, Cristian García-Romero, Taja Kuzman, Nikola Ljubešić, Rik Van Noord, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Peter Rupnik, and 1 others. 2022. Macocu: Massive collection and curation of monolingual and bilingual data: focus on under-resourced la… [find](https://scholar.google.com/scholar?q=Marta+Ba%C3%B1%C3%B3n%2C+Miquel+Espl%C3%A0%2DGomis%2C+Mikel+L+Forcada%2C+Cristian+Garc%C3%ADa%2DRomero%2C+Taja+Kuzman%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+Rik+Van+Noord%2C+Leopoldo+Pla+Sempere%2C+Gema+Ram%C3%ADrez%2DS%C3%A1nchez%2C+Peter+Rupnik%2C+and+1+others%2E+2022%2E+Ma)
4. Vuk Batanović, Nikola Ljubešić, Tanja Samardžić, and Tomaž Erjavec. 2023. Serbian linguistic training corpus SETimes.SR 2.0. Slovenian language resource repository CLARIN.SI. [find](https://scholar.google.com/scholar?q=Vuk+Batanovic%CC%81%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+Tanja+Samard%C5%BEic%CC%81%2C+and+Toma%C5%BE+Erjavec%2E+2023%2E+Serbian+linguistic+training+corpus+SETimes%2ESR+2%2E0%2E+Slovenian+language+resource+repository+CLARIN%2ESI%2E)
5. Damir Ćavar and Dunja Brozović Rončević. 2012. Riznica: the croatian language corpus. Prace filologiczne, 63:51–65. [find](https://scholar.google.com/scholar?q=Damir+C%CC%81avar+and+Dunja+Brozovic%CC%81+Ronc%CC%8Cevic%CC%81%2E+2012%2E+Riznica%3A+the+croatian+language+corpus%2E+Prace+filologiczne%2C+63%3A51%E2%80%9365%2E)
6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
7. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. [The commitmentbank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/). In Proceedings of Sinn und Bedeutung, volume 23, pages 107–124. Data can be found at [https://github.com/mcdm/CommitmentBank/](https://github.com/mcdm/CommitmentBank/). [↗](https://github.com/mcdm/CommitmentBank/)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
9. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International conference on machine learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
10. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
11. Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2023. Get to know your parallel data: Performing English variety and genre classification over MaCoCu corpora. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 91–103, Dubrovnik, Croatia. Association for Com… [find](https://scholar.google.com/scholar?q=Taja+Kuzman%2C+Peter+Rupnik%2C+and+Nikola+Ljube%C5%A1ic%CC%81%2E+2023%2E+Get+to+know+your+parallel+data%3A+Performing+English+variety+and+genre+classification+over+MaCoCu+corpora%2E+In+Tenth+Workshop+on+NLP+for+Similar+Lan)
12. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. KR, 2012(13th):3. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+KR%2C+2012%2813th%29%3A3%2E)
13. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and 1 others. 2020. [Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation](https://md.research.elamur.ai/free/en/papers/liang-2020-xglue-new-benchmark-dataset-cross/). arXiv preprint arXiv:2004.01401.
14. Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, and Tamás Váradi. 2024. Hulu: Hungarian language understanding benchmark kit. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluati… [find](https://scholar.google.com/scholar?q=No%C3%A9mi+Ligeti%2DNagy%2C+Gergo%CC%8B+Ferenczi%2C+Eniko%CC%8B+H%C3%A9ja%2C+L%C3%A1szl%C3%B3+J%C3%A1nos+Laki%2C+No%C3%A9mi+Vad%C3%A1sz%2C+Zijian+Gyo%CC%8Bzo%CC%8B+Yang%2C+and+Tam%C3%A1s+V%C3%A1radi%2E+2024%2E+Hulu%3A+Hungarian+language+understanding+benchmark+kit%2E+In+Proceedings+of+t)
15. Nikola Ljubešić. 2021. Text collection for training the BERTić transformer model BERTić-data. Slovenian language resource repository CLARIN.SI. [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81%2E+2021%2E+Text+collection+for+training+the+BERTic%CC%81+transformer+model+BERTic%CC%81%2Ddata%2E+Slovenian+language+resource+repository+CLARIN%2ESI%2E)
16. Nikola Ljubešić and Tomaž Erjavec. 2021. Montenegrin web corpus meWaC 1.0. Slovenian language resource repository CLARIN.SI. [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81+and+Toma%C5%BE+Erjavec%2E+2021%2E+Montenegrin+web+corpus+meWaC+1%2E0%2E+Slovenian+language+resource+repository+CLARIN%2ESI%2E)
17. Nikola Ljubešić and Filip Klubička. 2014. {bs,hr,sr}WaC - web corpora of Bosnian, Croatian and Serbian. In Proceedings of the 9th Web as Corpus Workshop (WaC-9), pages 29–35, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81+and+Filip+Klubic%CC%8Cka%2E+2014%2E+%7Bbs%2Chr%2Csr%7DWaC+%2D+web+corpora+of+Bosnian%2C+Croatian+and+Serbian%2E+In+Proceedings+of+the+9th+Web+as+Corpus+Workshop+%28WaC%2D9%29%2C+pages+29%E2%80%9335%2C+Gothenburg%2C+Sweden%2E+Ass)
18. Nikola Ljubešić and Taja Kuzman. 2024. Classla-web: comparable web corpora of south slavic languages enriched with linguistic and genre annotation. arXiv preprint arXiv:2403.12721. [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81+and+Taja+Kuzman%2E+2024%2E+Classla%2Dweb%3A+comparable+web+corpora+of+south+slavic+languages+enriched+with+linguistic+and+genre+annotation%2E+arXiv+preprint+arXiv%3A2403%2E12721%2E)
19. Nikola Ljubešić and Davor Lauc. 2021. BERTić – the transformer language model for Bosnian, Croatian, Montenegrin and Serbian. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 37–42. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81+and+Davor+Lauc%2E+2021%2E+BERTic%CC%81+%E2%80%93+the+transformer+language+model+for+Bosnian%2C+Croatian%2C+Montenegrin+and+Serbian%2E+In+Proceedings+of+the+8th+Workshop+on+Balto%2DSlavic+Natural+Language+Proc)
20. Nikola Ljubešić, Mirjana Starović, Taja Kuzman, and Tanja Samardžić. 2022. Choice of plausible alternatives dataset in serbian COPA-SR. Slovenian language resource repository CLARIN.SI. [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81%2C+Mirjana+Starovic%CC%81%2C+Taja+Kuzman%2C+and+Tanja+Samard%C5%BEic%CC%81%2E+2022%2E+Choice+of+plausible+alternatives+dataset+in+serbian+COPA%2DSR%2E+Slovenian+language+resource+repository+CLARIN%2ESI%2E)
21. Nikola Ljubešić, Vít Suchomel, Peter Rupnik, Taja Kuzman, and Rik van Noord. 2024a. Language models on a diet: Cost-efficient development of encoders for closely-related languages via additional pretraining. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced L… [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81%2C+V%C3%ADt+Suchomel%2C+Peter+Rupnik%2C+Taja+Kuzman%2C+and+Rik+van+Noord%2E+2024a%2E+Language+models+on+a+diet%3A+Cost%2Defficient+development+of+encoders+for+closely%2Drelated+languages+via+additional+pret)
22. Nikola Ljubešić, Vít Suchomel, Peter Rupnik, Taja Kuzman, and Rik van Noord. 2024b. Language models on a diet: Cost-efficient development of encoders for closely-related languages via additional pretraining. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced L… [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81%2C+V%C3%ADt+Suchomel%2C+Peter+Rupnik%2C+Taja+Kuzman%2C+and+Rik+van+Noord%2E+2024b%2E+Language+models+on+a+diet%3A+Cost%2Defficient+development+of+encoders+for+closely%2Drelated+languages+via+additional+pret)
23. Nikola Ljubešić, Filip Klubička, Željko Agić, and Ivo-Pavao Jazbec. 2016. New inflectional lexicons and training corpora for improved morphosyntactic annotation of croatian and serbian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, F… [find](https://scholar.google.com/scholar?q=Nikola+Ljube%C5%A1ic%CC%81%2C+Filip+Klubic%CC%8Cka%2C+%C5%BDeljko+Agic%CC%81%2C+and+Ivo%2DPavao+Jazbec%2E+2016%2E+New+inflectional+lexicons+and+training+corpora+for+improved+morphosyntactic+annotation+of+croatian+and+serbian%2E+In+Proceedi)
24. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. [mmBERT: A modern multilingual encoder with annealed language learning](https://md.research.elamur.ai/free/en/papers/marone-2025-mmbert-modern-multilingual-encoder-annealed/). Preprint, arXiv:2509.06888.
25. Ulfeta A Marovac, Aldina R Avdić, and Nikola Lj Milošević. 2023. A survey of resources and methods for natural language processing of serbian language. arXiv preprint arXiv:2304.05468. — [arXiv:2304.05468](https://arxiv.org/abs/2304.05468) [find](https://scholar.google.com/scholar?q=Ulfeta+A+Marovac%2C+Aldina+R+Avdic%CC%81%2C+and+Nikola+Lj+Milo%C5%A1evic%CC%81%2E+2023%2E+A+survey+of+resources+and+methods+for+natural+language+processing+of+serbian+language%2E+arXiv+preprint+arXiv%3A2304%2E05468%2E+%E2%80%94+%5BarXiv%3A2304)
26. Michal Mochtak, Peter Rupnik, and Nikola Ljubešić. 2024. The ParlaSent multilingual training dataset for sentiment identification in parliamentary proceedings. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024… [find](https://scholar.google.com/scholar?q=Michal+Mochtak%2C+Peter+Rupnik%2C+and+Nikola+Ljube%C5%A1ic%CC%81%2E+2024%2E+The+ParlaSent+multilingual+training+dataset+for+sentiment+identification+in+parliamentary+proceedings%2E+In+Proceedings+of+the+2024+Joint+Intern)
27. Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannhei… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%E2%80%99arez%2C+Benoit+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+Proceedings+of+the+Workshop+on+Challen)
28. Tomás Freitas Osório, Bernardo Leite, Henrique Lopes Cardoso, Luís Gomes, João Rodrigues, Rodrigo Santos, and António Branco. 2024. Portulan extraglue datasets and models: Kick-starting a benchmark for the neural processing of portuguese. In Proceedings of the 17th Workshop on Building and Using Com… [find](https://scholar.google.com/scholar?q=Tom%C3%A1s+Freitas+Os%C3%B3rio%2C+Bernardo+Leite%2C+Henrique+Lopes+Cardoso%2C+Lu%C3%ADs+Gomes%2C+Jo%C3%A3o+Rodrigues%2C+Rodrigo+Santos%2C+and+Ant%C3%B3nio+Branco%2E+2024%2E+Portulan+extraglue+datasets+and+models%3A+Kick%2Dstarting+a+benchmark+fo)
29. Edvin Pakoci, Branislav Popović, and Darko Pekar. 2019. Using morphological data in language modeling for serbian large vocabulary speech recognition. Computational intelligence and neuroscience, 2019(1):5072918. [find](https://scholar.google.com/scholar?q=Edvin+Pakoci%2C+Branislav+Popovic%CC%81%2C+and+Darko+Pekar%2E+2019%2E+Using+morphological+data+in+language+modeling+for+serbian+large+vocabulary+speech+recognition%2E+Computational+intelligence+and+neuroscience%2C+201)
30. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
31. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
32. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+and+1+others%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
33. Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. [Massively multilingual transfer for NER](https://md.research.elamur.ai/free/en/papers/rahimi-2019-massively-multilingual-transfer-ner/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
34. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+AAAI+spring+symposium%3A+logical+formalizations+of+c)
35. Peter Rupnik, Taja Kuzman, and Nikola Ljubešić. 2023. Benchić-lang: A benchmark for discriminating between bosnian, croatian, montenegrin and serbian. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 113–120. [find](https://scholar.google.com/scholar?q=Peter+Rupnik%2C+Taja+Kuzman%2C+and+Nikola+Ljube%C5%A1ic%CC%81%2E+2023%2E+Benchic%CC%81%2Dlang%3A+A+benchmark+for+discriminating+between+bosnian%2C+croatian%2C+montenegrin+and+serbian%2E+In+Tenth+Workshop+on+NLP+for+Similar+Languages%2C)
36. Tanja Samardžić, Mirjana Starović, Željko Agić, and Nikola Ljubešić. 2017. Universal Dependencies for Serbian in comparison with Croatian and other Slavic languages. In Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing, pages 39– 44, Valencia, Spain. Association for Com… [find](https://scholar.google.com/scholar?q=Tanja+Samard%C5%BEic%CC%81%2C+Mirjana+Starovic%CC%81%2C+%C5%BDeljko+Agic%CC%81%2C+and+Nikola+Ljube%C5%A1ic%CC%81%2E+2017%2E+Universal+Dependencies+for+Serbian+in+comparison+with+Croatian+and+other+Slavic+languages%2E+In+Proceedings+of+the+6th+Work)
37. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. Russiansuperglue: A russian language understanding evaluation benchmark. In Proceedings of the 2020 conference o… [find](https://scholar.google.com/scholar?q=Tatiana+Shavrina%2C+Alena+Fenogenova%2C+Emelyanov+Anton%2C+Denis+Shevelev%2C+Ekaterina+Artemova%2C+Valentin+Malykh%2C+Vladislav+Mikhailov%2C+Maria+Tikhonova%2C+Andrey+Chertok%2C+and+Andrey+Evlampiev%2E+2020%2E+Russiansuper)
38. Mihailo Škorić. 2024a. Novi jezički modeli za srpski jezik. Infoteka, 24. ArXiv:2402.14379. — [arXiv:2402.14379](https://arxiv.org/abs/2402.14379) [find](https://scholar.google.com/scholar?q=Mihailo+%C5%A0koric%CC%81%2E+2024a%2E+Novi+jezic%CC%8Cki+modeli+za+srpski+jezik%2E+Infoteka%2C+24%2E+ArXiv%3A2402%2E14379%2E+%E2%80%94+%5BarXiv%3A2402%2E14379%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E14379%29)
39. Mihailo Škorić. 2024b. Novi jezički modeli za srpski jezik. Infoteka, 24. [find](https://scholar.google.com/scholar?q=Mihailo+%C5%A0koric%CC%81%2E+2024b%2E+Novi+jezic%CC%8Cki+modeli+za+srpski+jezik%2E+Infoteka%2C+24%2E)
40. Mihailo Škorić and Nikola Janković. 2024. New textual corpora for serbian language modeling. arXiv preprint arXiv:2405.09250. — [arXiv:2405.09250](https://arxiv.org/abs/2405.09250) [find](https://scholar.google.com/scholar?q=Mihailo+%C5%A0koric%CC%81+and+Nikola+Jankovic%CC%81%2E+2024%2E+New+textual+corpora+for+serbian+language+modeling%2E+arXiv+preprint+arXiv%3A2405%2E09250%2E+%E2%80%94+%5BarXiv%3A2405%2E09250%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E09250%29)
41. Ranka Stanković, Cvetana Krstev, Branislava Šandrih Todorović, Dusko Vitas, Mihailo Skoric, and Milica Ikonić Nešić. 2022. Distant reading in digital humanities: Case study on the Serbian part of the EL-TeC collection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference… [find](https://scholar.google.com/scholar?q=Ranka+Stankovic%CC%81%2C+Cvetana+Krstev%2C+Branislava+%C5%A0andrih+Todorovic%CC%81%2C+Dusko+Vitas%2C+Mihailo+Skoric%2C+and+Milica+Ikonic%CC%81+Ne%C5%A1ic%CC%81%2E+2022%2E+Distant+reading+in+digital+humanities%3A+Case+study+on+the+Serbian+part+of+)
42. Ranka Stankovic, Branislava Šandrih, Cvetana Krstev, Miloš Utvić, and Mihailo Skoric. 2020. Machine learning and deep neural network-based lemmatization and morphosyntactic tagging for Serbian. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3954–3962, Marseille, F… [find](https://scholar.google.com/scholar?q=Ranka+Stankovic%2C+Branislava+%C5%A0andrih%2C+Cvetana+Krstev%2C+Milo%C5%A1+Utvic%CC%81%2C+and+Mihailo+Skoric%2E+2020%2E+Machine+learning+and+deep+neural+network%2Dbased+lemmatization+and+morphosyntactic+tagging+for+Serbian%2E+In+Pr)
43. European Language Resources Association. and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32.
44. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
45. Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. 2023. [Superclue: A comprehensive chinese large language model benchmark](https://md.research.elamur.ai/free/en/papers/xu-2023-superclue-comprehensive-chinese-large/). arXiv preprint arXiv:2307.15020.
46. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, and 1 others. 2025. [Mmlu-prox: A multilingual benchmark for advanced large language model evaluation](https://md.research.elamur.ai/free/en/papers/xuan-2025-mmlu-prox-multilingual-benchmark-advanced/). arXiv preprint arXiv:2503.10497.
47. Aleš Žagar and Marko Robnik-Šikonja. 2022. Slovene superglue benchmark: translation and evaluation. arXiv preprint arXiv:2202.04994. [find](https://scholar.google.com/scholar?q=Ale%C5%A1+%C5%BDagar+and+Marko+Robnik%2D%C5%A0ikonja%2E+2022%2E+Slovene+superglue+benchmark%3A+translation+and+evaluation%2E+arXiv+preprint+arXiv%3A2202%2E04994%2E)
48. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. [Record: Bridging the gap between human and machine commonsense reading comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/). arXiv preprint arXiv:1810.12885.
49. Minh-Nam Tran, Phu-Vinh Nguyen, Long Nguyen, and Dien Dinh. 2024. ViGLUE: A Vietnamese general language understanding benchmark and analysis of Vietnamese language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4174–4189, Mexico City, Mexico. Association for… [find](https://scholar.google.com/scholar?q=Minh%2DNam+Tran%2C+Phu%2DVinh+Nguyen%2C+Long+Nguyen%2C+and+Dien+Dinh%2E+2024%2E+ViGLUE%3A+A+Vietnamese+general+language+understanding+benchmark+and+analysis+of+Vietnamese+language+models%2E+In+Findings+of+the+Associati)
50. Matej Ulčar and Marko Robnik-Šikonja. 2020. FinEst BERT and CroSloEngual BERT: Less is more in multilingual models. In Text, Speech, and Dialogue (TSD 2020), volume 12284 of Lecture Notes in Computer Science, pages 104–118. Springer. [find](https://scholar.google.com/scholar?q=Matej+Ulc%CC%8Car+and+Marko+Robnik%2D%C5%A0ikonja%2E+2020%2E+FinEst+BERT+and+CroSloEngual+BERT%3A+Less+is+more+in+multilingual+models%2E+In+Text%2C+Speech%2C+and+Dialogue+%28TSD+2020%29%2C+volume+12284+of+Lecture+Notes+in+Computer)
51. Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2022. Basqueglue: A natural language understanding benchmark for basque. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1603– 1612. [find](https://scholar.google.com/scholar?q=Gorka+Urbizu%2C+I%C3%B1aki+San+Vicente%2C+Xabier+Saralegi%2C+Rodrigo+Agerri%2C+and+Aitor+Soroa%2E+2022%2E+Basqueglue%3A+A+natural+language+understanding+benchmark+for+basque%2E+In+Proceedings+of+the+Thirteenth+Language+Re)
52. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C)

*Updated 31 October 2026*