--- title: "GreekMMLU — paper by Zhang et al. (2026)" description: "Greek is often present in the training data of multilingual LLMs, but there are few trustworthy ways to… 1 reference, cited by 1 paper here." authors: ["Yang Zhang", "Mersin Konomi", "Christos Xypolopoulos", "Konstantinos Divriotis", "Konstantinos Skianis", "Giannis Nikolentzos", "Giorgos Stamou", "Guokan Shang", "Michalis Vazirgiannis"] year: 2026 venue: "Findings of the Association for Computational Linguistics: ACL 2026" source: "https://aclanthology.org/2026.findings-acl.448/" doi: "10.18653/v1/2026.findings-acl.448" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-greekmmlu-native-sourced-multitask-benchmark/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-greekmmlu-native-sourced-multitask-benchmark/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek # GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek **Authors:** [Yang Zhang](https://md.research.elamur.ai/free/en/authors/yang-zhang/), Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, [Konstantinos Skianis](https://md.research.elamur.ai/free/en/authors/konstantinos-skianis/), Giannis Nikolentzos, [Giorgos Stamou](https://md.research.elamur.ai/free/en/authors/giorgos-stamou/), [Guokan Shang](https://md.research.elamur.ai/free/en/authors/guokan-shang/) and [Michalis Vazirgiannis](https://md.research.elamur.ai/free/en/authors/michalis-vazirgiannis/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.448/) · [PDF](https://aclanthology.org/2026.findings-acl.448.pdf) · [DOI 10.18653/v1/2026.findings-acl.448](https://doi.org/10.18653/v1/2026.findings-acl.448) · [OpenAlex](https://openalex.org/W7166799031) **Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read ## Summary Greek is often present in the training data of multilingual LLMs, but there are few trustworthy ways to measure how well these models handle it. Most existing tests are machine translations of English datasets, which carry translation artifacts and US-centric cultural assumptions. The authors address this with GreekMMLU, a multiple-choice benchmark built only from questions originally written in Greek. The questions come from openly licensed school, university, vocational, civil-service, citizenship and driving examinations. They are organized into a new taxonomy of 45 subjects and labeled by educational level. Part of the data is held back for a private leaderboard to limit test contamination. The team tested more than 80 open and proprietary models. Commercial frontier systems clearly lead open-weight ones, and accuracy rises with model size. Topics tied to Greek culture are hardest for generic multilingual models, while Greek- and Europe-focused models handle them better. The resource gives Greek NLP a native, culturally grounded yardstick. It is also a template for other under-resourced languages, Ukrainian among them. ## Key points - **Size and structure:** 21,805 questions across 45 subjects, grouped into STEM, Humanities, Social Sciences and Other. - Eight subjects are specifically Greek, such as Greek History, Greek Mythology and Greek Traditions. - Each question has 2–4 options with exactly one correct answer. - Difficulty levels run from primary school to professional exams. - **Public and private splits:** 16,857 items are publicly released and 4,948 are reserved for a hidden leaderboard hosted on Hugging Face Spaces. - **Scale and prompting:** Models below roughly 2B parameters perform near chance. Five-shot prompting helps mid-size and large models, notably the Gemma and Llama families, but not the smallest ones. - **Greek-adapted models:** Llama-Krikri-8B, Meltemi and EuroLLM are compared with general models such as Qwen, Gemma, GPT-5.2 and Gemini 3 Flash. The Greek-adapted models show consistent advantages on culturally specific subjects. - **Calibration:** The correlation between confidence and accuracy is strongest for Llama-Krikri-8B (r = 0.93). It is weaker for Qwen2.5-7B (r = 0.74) and very poor for Llama-2-7B (r = 0.13). Question length barely affects model confidence. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 138 works · 6,590 citations - [École Polytechnique](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique/) — University, Palaiseau, France · 15 works · 6,688 citations - [National Technical University of Athens](https://md.research.elamur.ai/free/en/universities/national-technical-university-of-athens/) — University, Athens, Greece · 8 works · 461 citations - [Laboratoire d'Informatique de l'École Polytechnique](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-de-l-ecole-polytechnique/) — Research institute, Palaiseau, France · 3 works · 105 citations - [University of Ioannina](https://md.research.elamur.ai/free/en/universities/university-of-ioannina/) — University, Ioannina, Greece · 2 works · 2 citations Also on the byline: [University of Peloponnese](https://ror.org/04d4d3c02). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works ## Cite Cite as: Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang and Michalis Vazirgiannis. 2026. GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek. Findings of the Association for Computational Linguistics: ACL 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhang2026greekmmlu, title = {{GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek}}, author = {Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang and Michalis Vazirgiannis}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, year = {2026}, doi = {10.18653/v1/2026.findings-acl.448}, url = {https://doi.org/10.18653/v1/2026.findings-acl.448} } ``` - [← previous GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/zhang-2026-gm-prm-generative-multimodal-process/) - [next → GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control](https://md.research.elamur.ai/free/en/papers/simoni-2026-gtpo-stabilizing-group-relative-policy/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. *Updated 2 November 2026*