---
title: "TurkBench — paper by Toraman et al. (2026)"
description: "It offers the Turkish NLP community a diagnostic tool with a public leaderboard. 35 references, cited by 6 papers here."
authors: ["Cagri Toraman", "Ahmet Kaan Sever", "Ayşe Aysu Cengiz", "Elif Ecem Arslan", "Görkem Sevinç", "Sarp Kantar", "Mete Mert Birdal", "Yusuf Faruk Güldemir", "Ali Buğra Kanburoğlu", "Sezen Felekoğlu", "Birsen Şahin Kütük", "Büşra Tufan", "Elif Genç", "Serkan Coşkun", "Gupse Ekin Demir", "Muhammed Emin Arayıcı", "Olgun Dursun", "Onur Gungor", "Susan Üsküdarlı", "Abdullah Topraksoy", "Esra Darıcı"]
year: 2026
venue: "Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026)"
source: "https://aclanthology.org/2026.sigturk-1.12/"
doi: "10.18653/v1/2026.sigturk-1.12"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/toraman-2026-turkbench-benchmark-evaluating-turkish-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toraman-2026-turkbench-benchmark-evaluating-turkish-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / TurkBench: A Benchmark for Evaluating Turkish Large Language Models

# TurkBench: A Benchmark for Evaluating Turkish Large Language Models

**Authors:** [Cagri Toraman](https://md.research.elamur.ai/free/en/authors/cagri-toraman/), Ahmet Kaan Sever, Ayşe Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Sarp Kantar, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Olgun Dursun, [Onur Gungor](https://md.research.elamur.ai/free/en/authors/onur-gungor/), Susan Üsküdarlı, Abdullah Topraksoy and Esra Darıcı · **Year:** 2026 · **Venue:** [Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026)](https://md.research.elamur.ai/free/en/venues/sigturk/) ([Workshop Natural Language Processing for Turkic Languages 2026](https://md.research.elamur.ai/free/en/venues/sigturk/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.sigturk-1.12/) · [PDF](https://aclanthology.org/2026.sigturk-1.12.pdf) · [DOI 10.18653/v1/2026.sigturk-1.12](https://doi.org/10.18653/v1/2026.sigturk-1.12) · [OpenAlex](https://openalex.org/W7140146518)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/toraman-2026-turkbench-benchmark-evaluating-turkish-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM benchmarks are built around English. Translating them into Turkish misses the language's agglutinative morphology, its syntax, and its cultural knowledge. Earlier Turkish benchmarks mostly repackage public datasets, cover few skill types, and have no leaderboard that accepts online submissions. TurkBench addresses these gaps. It contains 8,151 newly curated items spread over 21 subtasks, which are grouped into six categories: Instruction Following, Knowledge, Reasoning, Language Understanding, Content Moderation, and Turkish Grammar and Vocabulary. The material comes from national exams, university coursework, sociology texts, literature and online platforms, obtained through partnerships with Turkish institutions. The authors did not adapt existing datasets or generate synthetic data, and experts checked every item for correctness, grammar and cultural sensitivity. Classification tasks are scored by accuracy, semantic similarity by Pearson and Spearman correlation, and open-ended tasks with an LLM judge. In the available results excerpt, model averages range from about 68 down to about 27, so the benchmark separates models clearly. It offers the Turkish NLP community a diagnostic tool with a public leaderboard.

## Key points
- **Size and coverage:** 8,151 instances across 21 subtasks. The largest are MMLU-style topics (2,373 items) and Instruction Following (997). The set also includes NER, Part-of-Speech, rare and loan words, metaphors and idioms, toxicity, bias, and two kinds of hallucination (truthfulness and faithfulness).
- **Quality validation:** human experts verified all items against three criteria (correctness, grammar and cultural sensitivity).
- **Evaluation method:** summarization, bias and instruction following are scored by GPT-4o-mini acting as judge. The authors acknowledge that the judge may carry its own biases.
- **Results excerpt:** Qwen3-235B averages 68.2 and TR-Gemma-9b averages 65.3. Small or Turkish-specific models such as TDM-8b-v0.1 (30.0) and Kumru-2B (27.3) score far lower.
- **Availability:** the leaderboard and submissions are hosted at huggingface.co/turkbench. Resources were contributed with support from OSYM, TUBITAK and Turkcell AI.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Middle East Technical University](https://md.research.elamur.ai/free/en/universities/middle-east-technical-university/) — University, Ankara, Türkiye · 9 works · 1,541 citations
- [Bilkent University](https://md.research.elamur.ai/free/en/universities/bilkent-university/) — University, Ankara, Türkiye · 5 works · 490 citations
- [Boğaziçi University](https://md.research.elamur.ai/free/en/universities/bogazici-university/) — University, Istanbul, Türkiye · 6 works · 400 citations
- [Hacettepe University](https://md.research.elamur.ai/free/en/universities/hacettepe-university/) — University, Ankara, Türkiye · 2 works · 8 citations
- [Istanbul University](https://md.research.elamur.ai/free/en/universities/istanbul-university/) — University, Istanbul, Türkiye · 2 works · 8 citations

Also on the byline: [Turkcell (Turkey)](https://ror.org/051vt5y84).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Cagri Toraman, Ahmet Kaan Sever, Ayşe Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Sarp Kantar, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Olgun Dursun, Onur Gungor, Susan Üsküdarlı, Abdullah Topraksoy and Esra Darıcı. 2026. TurkBench: A Benchmark for Evaluating Turkish Large Language Models. Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{toraman2026turkbench,
  title = {{TurkBench: A Benchmark for Evaluating Turkish Large Language Models}},
  author = {Cagri Toraman, Ahmet Kaan Sever, Ayşe Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Sarp Kantar, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Olgun Dursun, Onur Gungor, Susan Üsküdarlı, Abdullah Topraksoy and Esra Darıcı},
  booktitle = {Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.sigturk-1.12},
  url = {https://doi.org/10.18653/v1/2026.sigturk-1.12}
}
```

- [← previous Thinking beyond the anthropomorphic paradigm benefits LLM research](https://md.research.elamur.ai/free/en/papers/ibrahim-2026-thinking-beyond-anthropomorphic-paradigm-benefits/)
- [next → A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages](https://md.research.elamur.ai/free/en/papers/zhao-2026-comprehensive-evaluation-multilingual-chain-thought/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

35 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emre Can Acikgoz, Mete Erdogan, and Deniz Yuret. 2024. Bridging the bosphorus: Advancing Turkish large language models through strategies for low-resource language adaptation and benchmarking. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 242– 268, M… [find](https://scholar.google.com/scholar?q=Emre+Can+Acikgoz%2C+Mete+Erdogan%2C+and+Deniz+Yuret%2E+2024%2E+Bridging+the+bosphorus%3A+Advancing+Turkish+large+language+models+through+strategies+for+low%2Dresource+language+adaptation+and+benchmarking%2E+In+Proc)
2. Tayfun Ates, M Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, and Deniz Yuret. 2022. Craft: A benchmark for causal reasoning about forces and interactions. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2602–2627. [find](https://scholar.google.com/scholar?q=Tayfun+Ates%2C+M+Ates%CC%A7og%CC%86lu%2C+%C3%87ag%CC%86atay+Yig%CC%86it%2C+Ilker+Kesen%2C+Mert+Kobas%2C+Erkut+Erdem%2C+Aykut+Erdem%2C+Tilbe+Goksun%2C+and+Deniz+Yuret%2E+2022%2E+Craft%3A+A+benchmark+for+causal+reasoning+about+forces+and+interaction)
3. Irene Baucells, Javier Aula-Blasco, Iria de Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, José Javier Saiz, Robiert Sepúlveda-Torres, et al. 2025. [Iberobench: A benchmark for llm evaluation in iberian languages](https://md.research.elamur.ai/free/en/papers/baucells-2025-iberobench-benchmark-llm-evaluation-iberian/). In Proceedings of the 31st Internat…
4. M Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümüş, Banu Diri, Savaş Yıldırım, and Öner Aytaş. 2025. Tr-mmlu benchmark for large language models: Performance evaluation, challenges, and opportunities for improvement. In 2025 33rd Signal Processing and Communications Applications Conference (SIU), p… [find](https://scholar.google.com/scholar?q=M+Ali+Bayram%2C+Ali+Arda+Fincan%2C+Ahmet+Semih+G%C3%BCm%C3%BCs%CC%A7%2C+Banu+Diri%2C+Savas%CC%A7+Y%C4%B1ld%C4%B1r%C4%B1m%2C+and+%C3%96ner+Aytas%CC%A7%2E+2025%2E+Tr%2Dmmlu+benchmark+for+large+language+models%3A+Performance+evaluation%2C+challenges%2C+and+opportunities)
5. Mehmet Selman Baysan, Istanbul Bebek, and Tunga Güngör. 2025. Tr-mteb: A comprehensive benchmark and embedding model suite for turkish sentence representations. C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds), Findings of the Association for Computational Linguistics: EMNLP 2025, pag… [find](https://scholar.google.com/scholar?q=Mehmet+Selman+Baysan%2C+Istanbul+Bebek%2C+and+Tunga+G%C3%BCng%C3%B6r%2E+2025%2E+Tr%2Dmteb%3A+A+comprehensive+benchmark+and+embedding+model+suite+for+turkish+sentence+representations%2E+C%2E+Christodoulopoulos%2C+T%2E+Chakraborty%2C+)
6. Çağrı Çöltekin, A Seza Doğruöz, and Özlem Çetinoğlu. 2023. Resources for turkish natural language processing: A critical survey. Language Resources and Evaluation, 57(1):449–488. [find](https://scholar.google.com/scholar?q=%C3%87ag%CC%86r%C4%B1+%C3%87%C3%B6ltekin%2C+A+Seza+Dog%CC%86ru%C3%B6z%2C+and+%C3%96zlem+%C3%87etinog%CC%86lu%2E+2023%2E+Resources+for+turkish+natural+language+processing%3A+A+critical+survey%2E+Language+Resources+and+Evaluation%2C+57%281%29%3A449%E2%80%93488%2E)
7. Yakup Abrek Er, Ilker Kesen, Gözde Gül Şahin, and Aykut Erdem. 2025. Cetvel: A unified benchmark for evaluating language understanding, generation and cultural capacity of llms for turkish. arXiv preprint arXiv:2508.16431. [find](https://scholar.google.com/scholar?q=Yakup+Abrek+Er%2C+Ilker+Kesen%2C+G%C3%B6zde+G%C3%BCl+S%CC%A7ahin%2C+and+Aykut+Erdem%2E+2025%2E+Cetvel%3A+A+unified+benchmark+for+evaluating+language+understanding%2C+generation+and+cultural+capacity+of+llms+for+turkish%2E+arXiv+pre)
8. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. [Syntaxgym: An online platform for targeted evaluation of language models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76.
9. Dilek Z Hakkani-Tür, Kemal Oflazer, and Gökhan Tür. 2002. Statistical morphological disambiguation for agglutinative languages. Computers and the Humanities, 36(4):381–410. [find](https://scholar.google.com/scholar?q=Dilek+Z+Hakkani%2DT%C3%BCr%2C+Kemal+Oflazer%2C+and+G%C3%B6khan+T%C3%BCr%2E+2002%2E+Statistical+morphological+disambiguation+for+agglutinative+languages%2E+Computers+and+the+Humanities%2C+36%284%29%3A381%E2%80%93410%2E)
10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). CoRR, abs/2009.03300.
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
12. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International conference on machine learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
13. Wen Lai, Mohsen Mesgar, and Alexander Fraser. 2024. LLMs beyond English: Scaling the multilingual capability of LLMs with cross-lingual feedback. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8186–8213, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wen+Lai%2C+Mohsen+Mesgar%2C+and+Alexander+Fraser%2E+2024%2E+LLMs+beyond+English%3A+Scaling+the+multilingual+capability+of+LLMs+with+cross%2Dlingual+feedback%2E+In+Findings+of+the+Association+for+Computational+Lingu)
14. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
15. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+Xiaodong+Fan%2C+Ruofei+Zhang%2C+Rahul+Agrawal%2C+Edward+Cui%2C+Sining+Wei%2C+Taroon+Bharti%2C+)
16. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958.
17. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
18. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [Xcopa: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/).
19. Ali Safaya, Emirhan Kurtuluş, Arda Goktogan, and Deniz Yuret. 2022. Mukayese: Turkish NLP strikes back. In Findings of the Association for Computational Linguistics: ACL 2022, pages 846–863, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ali+Safaya%2C+Emirhan+Kurtulus%CC%A7%2C+Arda+Goktogan%2C+and+Deniz+Yuret%2E+2022%2E+Mukayese%3A+Turkish+NLP+strikes+back%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+846%E2%80%93863%2C+Dublin%2C+)
20. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
21. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Blog post, Stanford Center for Research on Foundation Models. Accessed: 2025-06-25. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2023%2E+Alpaca%3A+A+strong%2C+replicable+instruction%2Dfollowing+model%2E+Blog+pos)
22. Bernardo Magnini, Marco Madeddu, Michele Resta, Roberto Zanoli, Martin Cimmino, Paolo Albano, and Viviana Patti. 2025. A leaderboard for benchmarking llms on italian. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 636–646. [find](https://scholar.google.com/scholar?q=Bernardo+Magnini%2C+Marco+Madeddu%2C+Michele+Resta%2C+Roberto+Zanoli%2C+Martin+Cimmino%2C+Paolo+Albano%2C+and+Viviana+Patti%2E+2025%2E+A+leaderboard+for+benchmarking+llms+on+italian%2E+In+Proceedings+of+the+Eleventh+It)
23. Omer Nacar, Serry Taiseer Sibaee, Samar Ahmed, Safa Ben Atitallah, Adel Ammar, Yasser Alhabashi, Abdulrahman S Al-Batati, Arwa Alsehibani, Nour Qandos, Omar Elshehy, et al. 2025. [Towards inclusive arabic llms: A culturally aligned benchmark in arabic large language model evaluation](https://md.research.elamur.ai/free/en/papers/nacar-2025-inclusive-arabic-llms-culturally-aligned/). In Proceedings o…
24. Kemal Oflazer. 2014. Turkish and its challenges for language processing. Language resources and evaluation, 48(4):639–653. [find](https://scholar.google.com/scholar?q=Kemal+Oflazer%2E+2014%2E+Turkish+and+its+challenges+for+language+processing%2E+Language+resources+and+evaluation%2C+48%284%29%3A639%E2%80%93653%2E)
25. Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. 2024. Open Ko-LLM leaderboard: Evaluating large language models in Korean with Koh5 benchmark. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol… [find](https://scholar.google.com/scholar?q=Chanjun+Park%2C+Hyeonwoo+Kim%2C+Dahyun+Kim%2C+SeongHwan+Cho%2C+Sanghoon+Kim%2C+Sukyung+Lee%2C+Yungi+Kim%2C+and+Hwalsuk+Lee%2E+2024%2E+Open+Ko%2DLLM+leaderboard%3A+Evaluating+large+language+models+in+Korean+with+Koh5+benchm)
26. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
27. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. [Bbq: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). arXiv preprint arXiv:2110.08193.
28. Cagri Toraman. 2024. Adapting open-source generative large language models for low-resource languages: A case study for Turkish. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 30–44, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cagri+Toraman%2E+2024%2E+Adapting+open%2Dsource+generative+large+language+models+for+low%2Dresource+languages%3A+A+case+study+for+Turkish%2E+In+Proceedings+of+the+Fourth+Workshop+on+Multilingual+Representation+Le)
29. Elif Ecem Umutlu, Ayse Aysu Cengiz, Ahmet Kaan Sever, Seyma Erdem, Burak Aytan, Busra Tufan, Abdullah Topraksoy, Esra Darıcı, and Cagri Toraman. 2025. Evaluating the quality of benchmark datasets for low-resource languages: A case study on Turkish. In Proceedings of the Fourth Workshop on Generation… [find](https://scholar.google.com/scholar?q=Elif+Ecem+Umutlu%2C+Ayse+Aysu+Cengiz%2C+Ahmet+Kaan+Sever%2C+Seyma+Erdem%2C+Burak+Aytan%2C+Busra+Tufan%2C+Abdullah+Topraksoy%2C+Esra+Dar%C4%B1c%C4%B1%2C+and+Cagri+Toraman%2E+2025%2E+Evaluating+the+quality+of+benchmark+datasets+for+)
30. Arda Uzunoğlu and Gözde Gül Şahin. 2023. Benchmarking procedural language understanding for low-resource languages: A case study on turkish. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Associatio… [find](https://scholar.google.com/scholar?q=Arda+Uzunog%CC%86lu+and+G%C3%B6zde+G%C3%BCl+S%CC%A7ahin%2E+2023%2E+Benchmarking+procedural+language+understanding+for+low%2Dresource+languages%3A+A+case+study+on+turkish%2E+In+Proceedings+of+the+13th+International+Joint+Conference)
31. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
33. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. [Blimp: The benchmark of linguistic minimal pairs for english](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377–392.
34. Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Senel, Anna Korhonen, and Hinrich Schütze. 2024. Turkishmmlu: Measuring massive multitask language understanding in turkish. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7035–7055. [find](https://scholar.google.com/scholar?q=Arda+Y%C3%BCksel%2C+Abdullatif+K%C3%B6ksal%2C+L%C3%BCtfi+Kerem+Senel%2C+Anna+Korhonen%2C+and+Hinrich+Sch%C3%BCtze%2E+2024%2E+Turkishmmlu%3A+Measuring+massive+multitask+language+understanding+in+turkish%2E+In+Findings+of+the+Association+)
35. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in neural information processing systems, 36:46595–46623.

*Updated 1 November 2026*