---
title: "AMALIA — paper by Simplício et al. (2026) — summary and ideas"
description: "Alongside the model, they release an evaluation suite with translated standard tasks and four new… 73 references, cited by 1 paper here."
authors: ["Afonso Simplício", "Gonçalo Vinagre", "Miguel Moura Ramos", "Diogo Tavares", "Rafael Ferreira", "Giuseppe Attanasio", "Duarte M. Alves", "Inês Calvo", "Inês Vieira", "Rui Guerra", "James Furtado", "Beatriz Canaverde", "Iago Paulo", "Vasco Ramos", "Diogo Glória-Silva", "Miguel Faria", "Marcos Treviso", "Daniel Gomes", "Pedro Gomes", "David Semedo", "André Martins", "João Magalhães"]
year: 2026
venue: "Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1"
source: "https://aclanthology.org/2026.propor-1.38/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/simplicio-2026-amalia-fully-open-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/simplicio-2026-amalia-fully-open-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / AMALIA: A Fully Open Large Language Model for European Portuguese

# AMALIA: A Fully Open Large Language Model for European Portuguese

**Authors:** [Afonso Simplício](https://md.research.elamur.ai/free/en/authors/afonso-simplicio/), [Gonçalo Vinagre](https://md.research.elamur.ai/free/en/authors/goncalo-vinagre/), [Miguel Moura Ramos](https://md.research.elamur.ai/free/en/authors/miguel-moura-ramos/), [Diogo Tavares](https://md.research.elamur.ai/free/en/authors/diogo-tavares/), [Rafael Ferreira](https://md.research.elamur.ai/free/en/authors/rafael-ferreira/), [Giuseppe Attanasio](https://md.research.elamur.ai/free/en/authors/giuseppe-attanasio/), [Duarte M. Alves](https://md.research.elamur.ai/free/en/authors/duarte-m-alves/), [Inês Calvo](https://md.research.elamur.ai/free/en/authors/ines-calvo/), [Inês Vieira](https://md.research.elamur.ai/free/en/authors/ines-vieira/), Rui Guerra, [James Furtado](https://md.research.elamur.ai/free/en/authors/james-furtado/), Beatriz Canaverde, Iago Paulo, Vasco Ramos, [Diogo Glória-Silva](https://md.research.elamur.ai/free/en/authors/diogo-gloria-silva/), Miguel Faria, [Marcos Treviso](https://md.research.elamur.ai/free/en/authors/marcos-treviso/), Daniel Gomes, Pedro Gomes, [David Semedo](https://md.research.elamur.ai/free/en/authors/david-semedo/), [André Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) and [João Magalhães](https://md.research.elamur.ai/free/en/authors/joao-magalhaes/) · **Year:** 2026 · **Venue:** [Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1](https://md.research.elamur.ai/free/en/venues/propor/) ([International Conference on Computational Processing of Portuguese 2026](https://md.research.elamur.ai/free/en/venues/propor/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.propor-1.38/) · [PDF](https://aclanthology.org/2026.propor-1.38.pdf)

**Citations:** 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open LLMs are trained mostly on English, and European Portuguese (pt-PT) gets little attention. Its evaluation also tends to use machine-translated benchmarks that miss the variety's linguistic and cultural traits, and models often drift toward Brazilian Portuguese (pt-BR). AMALIA addresses this as a fully open model built on EuroLLM-9B. The authors altered EuroLLM's final pretraining phase to add filtered pt-PT web text from Arquivo.pt and long-context material. They then applied supervised fine-tuning and DPO preference tuning on a curated mix of pt-PT and English data. Alongside the model, they release an evaluation suite with translated standard tasks and four new datasets. These cover pt-PT generation, linguistic competence and pt-PT/pt-BR bias. A national high-school exam benchmark is also included. AMALIA performs about as well as comparable open models on translated benchmarks. It does clearly better on the native pt-PT tests and leads on pt-PT generation fidelity, while keeping attack success rates low on safety tests. The work shows that focused data and native evaluation can serve a lower-resource language variety.

## Key points
- **Arquivo.pt pretraining data:** 195 TB of raw WARC archives were processed with a Datatrove pipeline modeled on FineWeb2. The pipeline dropped `.br` domains, used Trafilatura extraction, applied heuristic filters, MinHash dedup and EuroFilter quality labels. The high- and medium-quality splits total 5.8B tokens.
- **Pretraining mixture:** the original 40B-token EuroLLM mixture was combined with 60B extra tokens (stronger code coverage) and 1.4B synthetic long-context tokens.
- **Post-training data:** covers instruction following, conversational reasoning, math and safety. It draws on PersonaHub/Nemotron-persona synthetic data generated with Gemma 3-27B and a pt-PT MT model, filtered with the DEITA scorer (threshold ≥5). It also includes a 200-entry Portuguese Linguistic Instructions set curated by an expert.
- **P3B3 benchmark:** most models favor pt-BR even when asked for pt-PT. Only AMALIA, EuroLLM and Gemma write pt-PT reliably, and AMALIA scores highest, judged by Gemini 2.5 Pro.
- **Linguistic competence and safety:** on the linguistic benchmark, AMALIA-DPO is the best fully open model, with top results in Lexicology and Culture-bound Semantics. On safety benchmarks scored by Qwen3Guard-8B, DPO more than halved the SFT model's attack success rate.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins and João Magalhães. 2026. AMALIA: A Fully Open Large Language Model for European Portuguese. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{simplicio2026amalia,
  title = {{AMALIA: A Fully Open Large Language Model for European Portuguese}},
  author = {Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins and João Magalhães},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.38/}
}
```

- [← previous Almost Clinical: Linguistic properties of synthetic electronic health records](https://md.research.elamur.ai/free/en/papers/sharoff-2026-almost-clinical-linguistic-properties-synthetic/)
- [next → AMATI at BEA 2026 Shared Task 2: Automatic Short Answer Grading with Inductive Logic Programming and a Large Language Model](https://md.research.elamur.ai/free/en/papers/willis-2026-amati-bea-2026-shared-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/)

## References

73 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loubna Ben Allal, Anton Lozhkov, and 20 others. 2025. [Smollm2: When smol goes big - data-centric training of a small language model](https://md.research.elamur.ai/free/en/papers/allal-2025-smollm2-smol-goes-big-data/). ArXiv, abs/2502.02737.
2. Project Apertus. 2025. [Apertus: Democratizing open and compliant llms for global language environments](https://md.research.elamur.ai/free/en/papers/cano-2026-apertus-democratizing-open-compliant-llms/). CoRR, abs/2509.14233.
3. Giuseppe Attanasio, Pierpaolo Basile, and 9 others. 2024. CALAMITA: challenge the abilities of language models in italian. In CLiC-it 2024, volume 3878 of CEUR Workshop Proceedings. CEUR-WS.org. [find](https://scholar.google.com/scholar?q=Giuseppe+Attanasio%2C+Pierpaolo+Basile%2C+and+9+others%2E+2024%2E+CALAMITA%3A+challenge+the+abilities+of+language+models+in+italian%2E+In+CLiC%2Dit+2024%2C+volume+3878+of+CEUR+Workshop+Proceedings%2E+CEUR%2DWS%2Eorg%2E)
4. Elie Bakouch, Loubna Ben Allal, and 21 others. 2025. SmolLM3: smol, multilingual, long-context reasoner. [https://huggingface.co/blog/smollm3](https://huggingface.co/blog/smollm3). [↗](https://huggingface.co/blog/smollm3)
5. Adrien Barbaresi. 2021. [Trafilatura: A web scraping library and command-line tool for text discovery and extraction](https://md.research.elamur.ai/free/en/papers/barbaresi-2021-trafilatura-web-scraping-library-command/). In ACL 2021 - System Demonstrations, pages 122–131. ACL.
6. Akhiad Bercovich, Itay Levy, and 131 others. 2025. [Llama-nemotron: Efficient reasoning models](https://md.research.elamur.ai/free/en/papers/bercovich-2025-llama-nemotron-efficient-reasoning-models/). Preprint, arXiv:2505.00949.
7. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. Preprint, arXiv:2308.09662. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj+and+Soujanya+Poria%2E+2023%2E+Red%2Dteaming+large+language+models+using+chain+of+utterances+for+safety%2Dalignment%2E+Preprint%2C+arXiv%3A2308%2E09662%2E)
8. Yonatan Bisk, Rowan Zellers, and 3 others. 2020. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In AAAI 2020, pages 7432–7439. AAAI Press.
9. A.Z. Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997, pages 21–29. [find](https://scholar.google.com/scholar?q=A%2EZ%2E+Broder%2E+1997%2E+On+the+resemblance+and+containment+of+documents%2E+In+Proceedings%2E+Compression+and+Complexity+of+SEQUENCES+1997%2C+pages+21%E2%80%9329%2E)
10. Peter Clark, Isaac Cowhey, and 5 others. 2018. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457.
11. Karl Cobbe, Vineet Kosaraju, and 10 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168.
12. Ganqu Cui, Lifan Yuan, and 7 others. 2023. Ultrafeedback: Boosting language models with high-quality feedback. Preprint, arXiv:2310.01377. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+and+7+others%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+Preprint%2C+arXiv%3A2310%2E01377%2E)
13. Amir Kargaran, Ayyoob Imani, and 2 others. 2023. [Glotlid: Language identification for low-resource languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/). In Findings of the Association for Computational Linguistics: EMNLP 2023, page 6155–6218. Association for Computational Linguistics.
14. Woosuk Kwon, Zhuohan Li, and 7 others. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In SOSP 2023, pages 611–626. ACM.
15. Andreas Köpf, Yannic Kilcher, and 16 others. 2023. [Openassistant conversations – democratizing large language model alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/). Preprint, arXiv:2304.07327.
16. DeepSeek-AI. 2025. Deepseek-v3.2-exp: Boosting long-context efficiency with deepseek sparse attention. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2025%2E+Deepseek%2Dv3%2E2%2Dexp%3A+Boosting+long%2Dcontext+efficiency+with+deepseek+sparse+attention%2E)
17. Seongyun Lee, Sue Hyun Park, and 2 others. 2024. Aligning to thousands of preferences via system message generalization. In Pluralistic Alignment Workshop at NeurIPS 2024. [find](https://scholar.google.com/scholar?q=Seongyun+Lee%2C+Sue+Hyun+Park%2C+and+2+others%2E+2024%2E+Aligning+to+thousands+of+preferences+via+system+message+generalization%2E+In+Pluralistic+Alignment+Workshop+at+NeurIPS+2024%2E)
18. Daniel Deutsch, Eleftheria Briakou, and 15 others. 2025. [Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects](https://md.research.elamur.ai/free/en/papers/deutsch-2025-wmt24-expanding-language-coverage-wmt24/). Preprint, arXiv:2502.12404.
19. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In ACL 2022, pages 3214–3252.
20. Abhimanyu Dubey, Abhinav Jauhri, and 9 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
21. Wei Liu, Weihao Zeng, and 3 others. 2023. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). Preprint, arXiv:2312.15685.
22. Dario Garcia-Gasulla, Adrián Tormos, and 5 others. 2025. Efficient safety retrofitting against jailbreaking for llms. In SAFECOMP 2025 Workshops, volume 15955 of Lecture Notes in Computer Science, pages 537–565. Springer. [find](https://scholar.google.com/scholar?q=Dario+Garcia%2DGasulla%2C+Adri%C3%A1n+Tormos%2C+and+5+others%2E+2025%2E+Efficient+safety+retrofitting+against+jailbreaking+for+llms%2E+In+SAFECOMP+2025+Workshops%2C+volume+15955+of+Lecture+Notes+in+Computer+Science%2C+pag)
23. Ricardo Lopes, João Magalhães, and David Semedo. 2024. Glória: A generative and open large language model for portuguese. In PROPOR 2024, Volume 1, pages 441–453. ACL. [find](https://scholar.google.com/scholar?q=Ricardo+Lopes%2C+Jo%C3%A3o+Magalh%C3%A3es%2C+and+David+Semedo%2E+2024%2E+Gl%C3%B3ria%3A+A+generative+and+open+large+language+model+for+portuguese%2E+In+PROPOR+2024%2C+Volume+1%2C+pages+441%E2%80%93453%2E+ACL%2E)
24. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
25. Tao Ge, Xin Chan, and 4 others. 2025. [Scaling synthetic data creation with 1,000,000,000 personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/). Preprint, arXiv:2406.20094.
26. Aitor Gonzalez-Agirre, Marc Pàmies, and 9 others. 2025. Salamandra technical report. arXiv preprint arXiv:2502.08489. — [arXiv:2502.08489](https://arxiv.org/abs/2502.08489) [find](https://scholar.google.com/scholar?q=Aitor+Gonzalez%2DAgirre%2C+Marc+P%C3%A0mies%2C+and+9+others%2E+2025%2E+Salamandra+technical+report%2E+arXiv+preprint+arXiv%3A2502%2E08489%2E+%E2%80%94+%5BarXiv%3A2502%2E08489%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E08489%29)
27. Yuling Gu, Oyvind Tafjord, and 4 others. 2025. OLMES: A standard for language model evaluations. In Findings of the ACL: NAACL 2025, pages 5005– 5033. ACL. [find](https://scholar.google.com/scholar?q=Yuling+Gu%2C+Oyvind+Tafjord%2C+and+4+others%2E+2025%2E+OLMES%3A+A+standard+for+language+model+evaluations%2E+In+Findings+of+the+ACL%3A+NAACL+2025%2C+pages+5005%E2%80%93+5033%2E+ACL%2E)
28. Dan Hendrycks, Collin Burns, and 5 others. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
29. Albert Jiang, Alexandre Abou Chahine, and Others. 2024. Ministral 8b instruct 2410. [https://huggingface.co/mistralai/](https://huggingface.co/mistralai/) Ministral-8B-Instruct-2410. [↗](https://huggingface.co/mistralai/)
30. Albert Q. Jiang, Alexandre Sablayrolles, and 16 others. 2023. Mistral 7b. CoRR, abs/2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+and+16+others%2E+2023%2E+Mistral+7b%2E+CoRR%2C+abs%2F2310%2E06825%2E)
31. Aishwarya Kamath, Johan Ferret, and 8 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Aishwarya+Kamath%2C+Johan+Ferret%2C+and+8+others%2E+2025%2E+Gemma+3+technical+report%2E+arXiv+preprint+arXiv%3A2503%2E19786%2E)
32. Anton Lozhkov, Raymond Li, and 55 others. 2024. Starcoder 2 and the stack v2: The next generation. CoRR, abs/2402.19173. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Raymond+Li%2C+and+55+others%2E+2024%2E+Starcoder+2+and+the+stack+v2%3A+The+next+generation%2E+CoRR%2C+abs%2F2402%2E19173%2E)
33. Pedro Henrique Martins, João Alves, and 15 others. 2025. [Eurollm-9b: Technical report](https://md.research.elamur.ai/free/en/papers/martins-2025-eurollm-9b-technical-report/). Preprint, arXiv:2506.04079.
34. Yev Meyer and Dane Corneil. 2025. Nemotron-Personas-USA: Synthetic personas aligned to real-world distributions. [find](https://scholar.google.com/scholar?q=Yev+Meyer+and+Dane+Corneil%2E+2025%2E+Nemotron%2DPersonas%2DUSA%3A+Synthetic+personas+aligned+to+real%2Dworld+distributions%2E)
35. Norman Mu, Jonathan Lu, and 2 others. 2024. A closer look at system message robustness. In Neurips Safe Generative AI Workshop 2024. [find](https://scholar.google.com/scholar?q=Norman+Mu%2C+Jonathan+Lu%2C+and+2+others%2E+2024%2E+A+closer+look+at+system+message+robustness%2E+In+Neurips+Safe+Generative+AI+Workshop+2024%2E)
36. Dhruv Nathawani, Shuoyang Ding, and 6 others. 2025a. Nemotron-Post-Training-Dataset-v2. [find](https://scholar.google.com/scholar?q=Dhruv+Nathawani%2C+Shuoyang+Ding%2C+and+6+others%2E+2025a%2E+Nemotron%2DPost%2DTraining%2DDataset%2Dv2%2E)
37. Dhruv Nathawani, Igor Gitman, and 6 others. 2025b. Nemotron-Post-Training-Dataset-v1. [find](https://scholar.google.com/scholar?q=Dhruv+Nathawani%2C+Igor+Gitman%2C+and+6+others%2E+2025b%2E+Nemotron%2DPost%2DTraining%2DDataset%2Dv1%2E)
38. NVIDIA. 2025. [Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model](https://md.research.elamur.ai/free/en/papers/basant-2025-nvidia-nemotron-nano-2-accurate/). Preprint, arXiv:2508.14444.
39. Team Olmo, Allyson Ettinger, and 66 others. 2025. [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/). Preprint, arXiv:2512.13961.
40. Team OLMo, Pete Walsh, and 38 others. 2024. 2 olmo 2 furious. ArXiv, abs/2501.00656. [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+and+38+others%2E+2024%2E+2+olmo+2+furious%2E+ArXiv%2C+abs%2F2501%2E00656%2E)
41. Guilherme Penedo, Hynek Kydlíček, and 3 others. 2024. Datatrove: large scale data processing. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Hynek+Kydl%C3%ADc%CC%8Cek%2C+and+3+others%2E+2024%2E+Datatrove%3A+large+scale+data+processing%2E)
42. Guilherme Penedo, Hynek Kydlíček, and 8 others. 2025. [Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language](https://md.research.elamur.ai/free/en/papers/penedo-2025-fineweb2-one-pipeline-scale-them/). Preprint, arXiv:2506.20920.
43. Irene Plaza, Nina Melero, and 5 others. 2024. Spanish and llm benchmarks: Is mmlu lost in translation? GACLM, pages 104–108. [find](https://scholar.google.com/scholar?q=Irene+Plaza%2C+Nina+Melero%2C+and+5+others%2E+2024%2E+Spanish+and+llm+benchmarks%3A+Is+mmlu+lost+in+translation%3F+GACLM%2C+pages+104%E2%80%93108%2E)
44. Jack W. Rae, Sebastian Borgeaud, and 78 others. 2022. Scaling language models: Methods, analysis & insights from training gopher. Preprint, arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+and+78+others%2E+2022%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+gopher%2E+Preprint%2C+arXiv%3A2112%2E11446%2E)
45. Rafael Rafailov, Archit Sharma, and 4 others. 2024. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Preprint, arXiv:2305.18290.
46. Hugo Sousa, Satya Almasian, and 2 others. 2025. Tradutor: Building a variety specific translation model. AAAI, 39(24):25183–25191. [find](https://scholar.google.com/scholar?q=Hugo+Sousa%2C+Satya+Almasian%2C+and+2+others%2E+2025%2E+Tradutor%3A+Building+a+variety+specific+translation+model%2E+AAAI%2C+39%2824%29%3A25183%E2%80%9325191%2E)
47. Aarohi Srivastava, Abhinav Rastogi, and 448 others. 2023. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/).
48. Trans. Mach. Learn. Res., 2023. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2023%2E)
49. Diogo Tavares, Rafael Ferreira, and 7 others. 2026. [PHEB: An european portuguese high school-level llm benchmark](https://md.research.elamur.ai/free/en/papers/tavares-2026-pheb-european-portuguese-high-school/). In Proceedings of the 2026 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2026). ELRA and ICCL.
50. Ryan Teknium, Jeffrey Quesnelle, and Chen Guang. 2024. Hermes 3 technical report. Preprint, arXiv:2408.11857. — [arXiv:2408.11857](https://arxiv.org/abs/2408.11857) [find](https://scholar.google.com/scholar?q=Ryan+Teknium%2C+Jeffrey+Quesnelle%2C+and+Chen+Guang%2E+2024%2E+Hermes+3+technical+report%2E+Preprint%2C+arXiv%3A2408%2E11857%2E+%E2%80%94+%5BarXiv%3A2408%2E11857%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E11857%29)
51. Klaudia Thellmann, Bernhard Stadler, and 9 others. 2024. Towards multilingual LLM evaluation for european languages. CoRR, abs/2410.08928. — [arXiv:2410.08928](https://arxiv.org/abs/2410.08928) [find](https://scholar.google.com/scholar?q=Klaudia+Thellmann%2C+Bernhard+Stadler%2C+and+9+others%2E+2024%2E+Towards+multilingual+LLM+evaluation+for+european+languages%2E+CoRR%2C+abs%2F2410%2E08928%2E+%E2%80%94+%5BarXiv%3A2410%2E08928%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E08928%29)
52. Miguel Moura Ramos, Duarte M. Alves, and 16 others. 2026. Eurollm-22b: Technical report. Preprint, arXiv:2602.05879. — [arXiv:2602.05879](https://arxiv.org/abs/2602.05879) [find](https://scholar.google.com/scholar?q=Miguel+Moura+Ramos%2C+Duarte+M%2E+Alves%2C+and+16+others%2E+2026%2E+Eurollm%2D22b%3A+Technical+report%2E+Preprint%2C+arXiv%3A2602%2E05879%2E+%E2%80%94+%5BarXiv%3A2602%2E05879%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2602%2E05879%29)
53. Bertie Vidgen, Hannah Rose Kirk, and 5 others. 2023. [Simplesafetytests: a test suite for identifying critical safety risks in large language models](https://md.research.elamur.ai/free/en/papers/vidgen-2023-simplesafetytests-test-suite-identifying-critical/). CoRR, abs/2311.08370.
54. Parker Riley, Timothy Dozat, and 6 others. 2023. [FRMT: A benchmark for few-shot region-aware machine translation](https://md.research.elamur.ai/free/en/papers/riley-2023-frmt-benchmark-few-shot-region/). Trans. Assoc. Comput. Linguistics, 11:671–685.
55. Inês Vieira, Inês Calvo, and 7 others. 2026. [ALBA: A european portuguese benchmark for evaluating language and linguistic dimensions in generative llms](https://md.research.elamur.ai/free/en/papers/vieira-2026-alba-european-portuguese-benchmark-evaluating/). In PROPOR 2026.
56. Morgane Rivière, Shreya Pathak, and 97 others. 2024. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118. [find](https://scholar.google.com/scholar?q=Morgane+Rivi%C3%A8re%2C+Shreya+Pathak%2C+and+97+others%2E+2024%2E+Gemma+2%3A+Improving+open+language+models+at+a+practical+size%2E+CoRR%2C+abs%2F2408%2E00118%2E)
57. Haoxiang Wang, Wei Xiong, and 3 others. 2024. [Interpretable preferences via multi-objective reward modeling and mixture-of-experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/). In Findings of the ACL: EMNLP 2024, pages 10582–10592. ACL.
58. Paul Röttger, Hannah Kirk, and 4 others. 2024. [Xstest: A test suite for identifying exaggerated safety behaviours in large language models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/). In NAACL 2024 (Volume 1: Long Papers), pages 5377–5400. ACL.
59. Rodrigo Santos, João Ricardo Silva, and 3 others. 2024. Advancing generative AI for Portuguese with open decoder gervásio PT*. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 16–26, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Rodrigo+Santos%2C+Jo%C3%A3o+Ricardo+Silva%2C+and+3+others%2E+2024%2E+Advancing+generative+AI+for+Portuguese+with+open+decoder+gerv%C3%A1sio+PT%2A%2E+In+Proceedings+of+the+3rd+Annual+Meeting+of+the+Special+Interest+Group+on)
60. Maarten Sap, Hannah Rashkin, and 3 others. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). CoRR, abs/1904.09728.
61. Afonso Simplício, Gonçalo Vinagre, and 20 others. 2026. AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese. [find](https://scholar.google.com/scholar?q=Afonso+Simpl%C3%ADcio%2C+Gon%C3%A7alo+Vinagre%2C+and+20+others%2E+2026%2E+AMALIA+Technical+Report%3A+A+Fully+Open+Source+Large+Language+Model+for+European+Portuguese%2E)
62. Shivalika Singh, Angelika Romanou, and 21 others. 2025. [Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation](https://md.research.elamur.ai/free/en/papers/singh-2025-global-mmlu-understanding-addressing-cultural/). In ACL 2025, pages 18761–18799. ACL.
63. Guijin Son, Hanwool Lee, and 7 others. 2025. [KMMLU: measuring massive multitask language understanding in korean](https://md.research.elamur.ai/free/en/papers/son-2025-kmmlu-measuring-massive-multitask-language/). In NAACL 2025, pages 4076–4104. ACL.
64. Minghao Wu, Weixuan Wang, and 8 others. 2025. The bitter lesson learned from 2,000+ multilingual benchmarks. ArXiv, abs/2504.15521. — [arXiv:2504.15521](https://arxiv.org/abs/2504.15521) [find](https://scholar.google.com/scholar?q=Minghao+Wu%2C+Weixuan+Wang%2C+and+8+others%2E+2025%2E+The+bitter+lesson+learned+from+2%2C000%2B+multilingual+benchmarks%2E+ArXiv%2C+abs%2F2504%2E15521%2E+%E2%80%94+%5BarXiv%3A2504%2E15521%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E15521%29)
65. Wenhan Xiong, Jingyu Liu, and 19 others. 2024. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). In NAACL 2024, pages 4643–4663. ACL.
66. Weihao Xuan, Rui Yang, and 16 others. 2025. [Mmlu-prox: A multilingual benchmark for advanced large language model evaluation](https://md.research.elamur.ai/free/en/papers/xuan-2025-mmlu-prox-multilingual-benchmark-advanced/). ArXiv, abs/2503.10497.
67. An Yang, Anfeng Li, and 58 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
68. An Yang, Baosong Yang, and 39 others. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+and+39+others%2E+2024%2E+Qwen2%2E5+technical+report%2E+CoRR%2C+abs%2F2412%2E15115%2E)
69. Ping Yu, Weizhe Yuan, and 5 others. 2025. R.i.p.: Better models by survival of the fittest prompts. Preprint, arXiv:2501.18578. — [arXiv:2501.18578](https://arxiv.org/abs/2501.18578) [find](https://scholar.google.com/scholar?q=Ping+Yu%2C+Weizhe+Yuan%2C+and+5+others%2E+2025%2E+R%2Ei%2Ep%2E%3A+Better+models+by+survival+of+the+fittest+prompts%2E+Preprint%2C+arXiv%3A2501%2E18578%2E+%E2%80%94+%5BarXiv%3A2501%2E18578%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E18578%29)
70. Haiquan Zhao, Chenhan Yuan, and 41 others. 2025. [Qwen3guard technical report](https://md.research.elamur.ai/free/en/papers/zhao-2025-qwen3guard-technical-report/). CoRR, abs/2510.14276.
71. Wenting Zhao, Xiang Ren, and 4 others. 2024. [Wildchat: 1m chatgpt interaction logs in the wild](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/). Preprint, arXiv:2405.01470.
72. Jeffrey Zhou, Tianjian Lu, and 6 others. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). Preprint, arXiv:2311.07911.
73. Andy Zou, Zifan Wang, and 2 others. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). CoRR, abs/2307.15043.

*Updated 1 November 2026*