--- title: "CLARIN-PT-LDB — paper by Silva et al. (2026)" description: "European Portuguese (PT-PT) has had no LLM leaderboard of its own. 3 references. Among them: 0." authors: ["João Ricardo Silva", "Luís Gomes", "António Branco"] year: 2026 venue: "Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1" source: "https://aclanthology.org/2026.propor-1.7/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/silva-2026-clarin-pt-ldb-open-llm/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/silva-2026-clarin-pt-ldb-open-llm/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility # CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility **Authors:** [João Ricardo Silva](https://md.research.elamur.ai/free/en/authors/joao-ricardo-silva/), [Luís Gomes](https://md.research.elamur.ai/free/en/authors/luis-gomes/) and [António Branco](https://md.research.elamur.ai/free/en/authors/antonio-branco/) · **Year:** 2026 · **Venue:** [Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1](https://md.research.elamur.ai/free/en/venues/propor/) ([International Conference on Computational Processing of Portuguese 2026](https://md.research.elamur.ai/free/en/venues/propor/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.propor-1.7/) · [PDF](https://aclanthology.org/2026.propor-1.7.pdf) · [OpenAlex](https://openalex.org/W7137146072) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary European Portuguese (PT-PT) has had no LLM leaderboard of its own. Existing Portuguese efforts mostly target the Brazilian variant, and no Portuguese benchmark measured safety safeguards or cultural alignment. The authors built CLARIN-PT-LDB, an open leaderboard hosted as a HuggingFace Space. Users submit open HuggingFace models, and the PORTULAN CLARIN infrastructure evaluates them on ten benchmarks: - **Tuguesice-PT**, a new culture benchmark written from scratch. - **DoNotAnswer-PT**, a curated translation of a safeguards dataset. - **Machine-translated reasoning and QA sets**: MuSR, MMLU, GPQA Diamond, MMLU Pro and AA-Omniscience-Public. - **CoPA, MRPC and RTE**, taken from ExtraGLUE. All tasks are scored generatively, so models must write their answer as text rather than have options ranked by logits. This works for any model, including API-only ones, and resembles real chatbot use. The pipeline runs on EleutherAI's evaluation harness with per-task YAML configurations, and every benchmark reports accuracy. Initial results cover Gervásio, Llama and Mistral at 8B, 24B and 70B. Scores rise with model size, and the harder MMLU Pro yields lower scores than MMLU. Gervásio, which was fine-tuned on Portuguese data, outscores its Llama base model on cultural alignment. ## Key points - **Tuguesice-PT**: - It has 327 short-answer question–answer pairs written by nine undergraduate annotators and checked by separate adjudicators. - Questions refer to Portugal only implicitly (e.g., asking about "the capital" without naming a country), so the test reveals the model's default cultural assumptions. - An answer counts as correct if it contains the gold answer (case-insensitive). - **DoNotAnswer-PT**: It has 939 harmful requests across 61 harm categories. A judge model sorts each response into one of six classes, such as refusal, disclaimer or compliance. - **Benchmark sizes**: MMLU has 14,042 items, MMLU Pro 12,032, AA-Omniscience-Public 600, GPQA Diamond 198, CoPA 500, MRPC 1,730 and RTE 3,000. - **Cultural alignment result**: On Tuguesice-PT, Gervásio 70B scores 39.76 against 25.69 for Llama 70B. At 8B, Gervásio scores 11.31 and Llama 9.79. - **Infrastructure**: The frontend is adapted from HuggingFace's Open LLM Leaderboard code. The backend runs on two NVIDIA L40S nodes (ten GPUs each) and two RTX Pro 6000 nodes (one 96 GB GPU each). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Lisbon](https://md.research.elamur.ai/free/en/universities/university-of-lisbon/) — University, Lisbon, Portugal · 26 works · 3,589 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works ## Cite Cite as: João Ricardo Silva, Luís Gomes and António Branco. 2026. CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{silva2026clarin, title = {{CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility}}, author = {João Ricardo Silva, Luís Gomes and António Branco}, booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1}, year = {2026}, url = {https://aclanthology.org/2026.propor-1.7/} } ``` - [← previous CLARIESG: An End-to-End System for ESG Analysis over Complex Tables in Corporate Reports](https://md.research.elamur.ai/free/en/papers/santacroce-2026-clariesg-end-end-system-esg/) - [next → CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems](https://md.research.elamur.ai/free/en/papers/sarwar-2026-clarity-framework-benchmark-conversational-language/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work. 1. Mohamad Alhajar. 2024. Open LLM Turkish leaderboard v0.2. [https://huggingface.co/spaces/](https://huggingface.co/spaces/) malhajar/OpenLLMTurkishLeaderboard. [↗](https://huggingface.co/spaces/) 2. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2024. [Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pape… 3. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open LLM leaderboard. [https://huggingface.co/spaces/open-llm-leaderboard-old/open_](https://huggingface.co/spaces/open-llm-leaderboard-old/open_) llm_leaderboard. \#\# Conclusion [↗](https://huggingface.co/spaces/open-llm-leaderboard-old/open_) *Updated 1 November 2026*