---
title: "IberoBench: A Benchmark for LLM Evaluation in Iberian Languages"
description: "Multi-task benchmarks are now the standard way to assess base LLMs, but they exist almost only for… 88 references, cited by 24 papers here."
authors: ["Irene Baucells", "Javier Aula-Blasco", "Iria de-Dios-Flores", "Silvia Paniagua Suárez", "Naiara Perez", "Anna Salles", "Susana Sotelo Docio", "Júlia Falcão", "Jose Javier Saiz", "Robiert Sepulveda Torres", "Jeremy Barnes", "Pablo Gamallo", "Aitor Gonzalez-Agirre", "German Rigau", "Marta Villegas"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.699/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/baucells-2025-iberobench-benchmark-llm-evaluation-iberian/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/baucells-2025-iberobench-benchmark-llm-evaluation-iberian/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / IberoBench: A Benchmark for LLM Evaluation in Iberian Languages

# IberoBench: A Benchmark for LLM Evaluation in Iberian Languages

**Authors:** Irene Baucells, [Javier Aula-Blasco](https://md.research.elamur.ai/free/en/authors/javier-aula-blasco/), Iria de-Dios-Flores, Silvia Paniagua Suárez, [Naiara Perez](https://md.research.elamur.ai/free/en/authors/naiara-perez/), Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, [Robiert Sepulveda Torres](https://md.research.elamur.ai/free/en/authors/robiert-sepulveda-torres/), [Jeremy Barnes](https://md.research.elamur.ai/free/en/authors/jeremy-barnes/), [Pablo Gamallo](https://md.research.elamur.ai/free/en/authors/pablo-gamallo/), [Aitor Gonzalez-Agirre](https://md.research.elamur.ai/free/en/authors/aitor-gonzalez-agirre/), [German Rigau](https://md.research.elamur.ai/free/en/authors/german-rigau/) and [Marta Villegas](https://md.research.elamur.ai/free/en/authors/marta-villegas/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.699/) · [PDF](https://aclanthology.org/2025.coling-main.699.pdf)

**Citations:** [24](https://md.research.elamur.ai/free/en/papers/baucells-2025-iberobench-benchmark-llm-evaluation-iberian/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multi-task benchmarks are now the standard way to assess base LLMs, but they exist almost only for English and a handful of other high-resource languages. IberoBench fills this gap for the official languages of Spain and Portugal, covering Catalan, Galician and Basque together with the European varieties of Portuguese and Spanish. The authors built it on EleutherAI's LM Evaluation Harness. They combined 40 previously published datasets with 22 newly created ones, all checked against common quality and reproducibility criteria. The tasks span ten categories, from commonsense reasoning and linguistic acceptability to mathematics, NLI, QA, summarization, translation and truthfulness. Using the benchmark, they tested 33 small and medium-sized base models, both monolingual and multilingual, in 0-shot and 5-shot settings. They also documented problems they found in datasets and in the Harness itself, and described how they mitigated them. The results show that models perform worse in Iberian languages than in English. Strong recent multilingual models give the most balanced performance across the five languages. The benchmark is public and gives a shared, reproducible basis for comparing models trained for these languages.

## Key points
- **Scale:** IberoBench has 62 tasks split into 179 subtasks, of which 22 tasks are new and 40 come from earlier work. It is integrated into the LM Evaluation Harness.
- **Evaluation:** 33 base LLMs were evaluated in 0-shot and 5-shot settings. These included BLOOM, XGLM, mGPT, Gemma, Llama 3-8B, Mistral-7B, OLMo, Falcon and Tower, plus language-specific models such as FLOR, Latxa and Carballo.
- **Results:** Among the multilingual models shown, Llama 3-8B, Mistral-7B and Gemma-7B score highest. Their Basque and Galician scores remain far below their Catalan, Spanish and Portuguese scores.
- **Hardest tasks:** Linguistic acceptability is among the most difficult task types.
- **Limitations:** BLEU and ROUGE are used for generative tasks and may not match human judgments. Bias, fairness and harmful-content tasks are planned but not yet included. The languages covered have over 510M native speakers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Irene Baucells, Javier Aula-Blasco, Iria de-Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, Robiert Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau and Marta Villegas. 2025. IberoBench: A Benchmark for LLM Evaluation in Iberian Languages. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{baucells2025iberobench,
  title = {{IberoBench: A Benchmark for LLM Evaluation in Iberian Languages}},
  author = {Irene Baucells, Javier Aula-Blasco, Iria de-Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, Robiert Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau and Marta Villegas},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.699/}
}
```

- [← previous Defense Against Prompt Injection Attack by Leveraging Attack Techniques](https://md.research.elamur.ai/free/en/papers/chen-2025-defense-against-prompt-injection-attack/)
- [next → Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/son-2025-linguistic-generalizability-test-time-scaling/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

88 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. PID2021[-128811OA-I00, PLEC2021-007662, and TED2021-130295B-C33, the latter also funded by the European Union Next Generation EU/PRTR This work has been funded by the grant ED431G-2023/04 by the Galician Ministry of Education, University and Professional Training, and the European Regional Developmen…](https://md.research.elamur.ai/free/en/papers/adelani-2025-irokobench-new-benchmark-african-languages/)
2. Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: the case for Basque. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4781–4788, Marseille, Franc… [find](https://scholar.google.com/scholar?q=Rodrigo+Agerri%2C+I%C3%B1aki+San+Vicente%2C+Jon+Ander+Campos%2C+Ander+Barrena%2C+Xabier+Saralegi%2C+Aitor+Soroa%2C+and+Eneko+Agirre%2E+2020%2E+Give+your+text+representation+models+some+love%3A+the+case+for+Basque%2E+In+Procee)
3. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al-This work is funded by the Ministerio para shamsi, Alessandro Cappelli, Ruxandra Cojocaru, la Transformación Digital y de la Función Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Pública and Plan de Recuperación, Transforma-Julien Launay, Quenti… — [The Falcon Series of Open Language Models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/)
4. Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. [Common voice: A massively-multilingual speech corpus](https://md.research.elamur.ai/free/en/papers/ardila-2019-common-voice-massively-multilingual-speech/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218–4222, Marseille, France. European Language Resources Association.
5. Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan.… [find](https://scholar.google.com/scholar?q=Jordi+Armengol%2DEstap%C3%A9%2C+Casimiro+Pio+Carrino%2C+Carlos+Rodriguez%2DPenagos%2C+Ona+de+Gibert+Bonet%2C+Carme+Armentano%2DOller%2C+Aitor+Gonzalez%2DAgirre%2C+Maite+Melero%2C+and+Marta+Villegas%2E+2021%2E+Are+multilingual+model)
6. Jordi Armengol-Estapé, Ona de Gibert Bonet, and Maite Melero. 2022. On the multilingual capabilities of very large-scale English language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3056–3068, Marseille, France. European Language Resources Association… [find](https://scholar.google.com/scholar?q=Jordi+Armengol%2DEstap%C3%A9%2C+Ona+de+Gibert+Bonet%2C+and+Maite+Melero%2E+2022%2E+On+the+multilingual+capabilities+of+very+large%2Dscale+English+language+models%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+an)
7. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
8. Javier Aula-Blasco, Júlia Falcão, Susana Sotelo Docio, Silvia Paniagua Suárez, Aitor Gonzalez-Agirre, and Marta Villegas. 2025. VeritasQA: A truthfulness benchmark aimed at multilingual transferability. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, Unit… [find](https://scholar.google.com/scholar?q=Javier+Aula%2DBlasco%2C+J%C3%BAlia+Falc%C3%A3o%2C+Susana+Sotelo+Docio%2C+Silvia+Paniagua+Su%C3%A1rez%2C+Aitor+Gonzalez%2DAgirre%2C+and+Marta+Villegas%2E+2025%2E+VeritasQA%3A+A+truthfulness+benchmark+aimed+at+multilingual+transferabilit)
9. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. [The belebele benchmark: a parallel reading comprehension dataset in 122 language variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/). Preprint, arXiv:2308.16884.
10. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
11. Jeanine Banks. 2024. Gemma: Introducing new state-of-the-art open models. [find](https://scholar.google.com/scholar?q=Jeanine+Banks%2E+2024%2E+Gemma%3A+Introducing+new+state%2Dof%2Dthe%2Dart+open+models%2E)
12. Nuria Bel, Marta Punsola, and Valle Ruíz-Fernández. 2024a. EsCoLA: Spanish corpus of linguistic acceptability. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6268–6277, Torino, Italia. ELRA and ICCL. [find](https://scholar.google.com/scholar?q=Nuria+Bel%2C+Marta+Punsola%2C+and+Valle+Ru%C3%ADz%2DFern%C3%A1ndez%2E+2024a%2E+EsCoLA%3A+Spanish+corpus+of+linguistic+acceptability%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computational+Linguistics%2C+L)
13. Núria Bel, Marta Punsola, and Valle Ruiz-Fernández. 2024b. CatCoLA: Catalan corpus of linguistic acceptability. Procesamiento del Lenguaje Natural, 73. [find](https://scholar.google.com/scholar?q=N%C3%BAria+Bel%2C+Marta+Punsola%2C+and+Valle+Ruiz%2DFern%C3%A1ndez%2E+2024b%2E+CatCoLA%3A+Catalan+corpus+of+linguistic+acceptability%2E+Procesamiento+del+Lenguaje+Natural%2C+73%2E)
14. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Lintang+Sutawika%2C+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Alham+Fikri+Aji%2C+Pawan+Sasanka+Ammanamanchi%2C+Sidney+Black%2C+Jordan+Clive%2C+Anthony+DiPofi%2C+Julen+Etxaniz%2C+Benja)
15. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2E+If+you+use+this+software%2C+please+cite+it+using+these)
16. Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model evaluation. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 339–352, Singapore. Association for Com… [find](https://scholar.google.com/scholar?q=Meriem+Boubdir%2C+Edward+Kim%2C+Beyza+Ermis%2C+Sara+Hooker%2C+and+Marzieh+Fadaee%2E+2023%2E+Elo+uncovered%3A+Robustness+and+best+practices+in+language+model+evaluation%2E+In+Proceedings+of+the+Third+Workshop+on+Natur)
17. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
18. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Trans. Intell. Syst. Technol., 15(3).
19. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
20. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
21. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semar… [find](https://scholar.google.com/scholar?q=Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+Wenzek%2C+Al+Yo)
22. Julen Etxaniz, Oscar Sainz, Naiara Perez, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. Latxa: An open language model and evaluation suite for basque. Preprint, arXiv:2403.20266. [find](https://scholar.google.com/scholar?q=Julen+Etxaniz%2C+Oscar+Sainz%2C+Naiara+Perez%2C+Itziar+Aldabe%2C+German+Rigau%2C+Eneko+Agirre%2C+Aitor+Ormazabal%2C+Mikel+Artetxe%2C+and+Aitor+Soroa%2E+2024%2E+Latxa%3A+An+open+language+model+and+evaluation+suite+for+basqu)
23. Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Denis Dimit… [find](https://scholar.google.com/scholar?q=Alena+Fenogenova%2C+Artem+Chervyakov%2C+Nikita+Martynov%2C+Anastasia+Kozlova%2C+Maria+Tikhonova%2C+Albina+Akhmetgareeva%2C+Anton+Emelyanov%2C+Denis+Shevelev%2C+Pavel+Lebedev%2C+Leonid+Sinev%2C+Ulyana+Isaeva%2C+Katerina+Kol)
24. Erick R. Fonseca, Leandro B. dos Santos, Marcelo Criscuolo, and Sandra M. Aluisio. 2016. Assin: Avaliação de similaridade semântica e inferência textual. In Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal, pages 13–15. [find](https://scholar.google.com/scholar?q=Erick+R%2E+Fonseca%2C+Leandro+B%2E+dos+Santos%2C+Marcelo+Criscuolo%2C+and+Sandra+M%2E+Aluisio%2E+2016%2E+Assin%3A+Avalia%C3%A7%C3%A3o+de+similaridade+sem%C3%A2ntica+e+infer%C3%AAncia+textual%2E+In+Computational+Processing+of+the+Portuguese+)
25. Severino Da Dalt, Joan Llop, Irene Baucells, Marc Pamies, Yishi Xu, Aitor Gonzalez-Agirre, and Marta Villegas. 2024. FLOR: On the effectiveness of language adaptation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLI… [find](https://scholar.google.com/scholar?q=Severino+Da+Dalt%2C+Joan+Llop%2C+Irene+Baucells%2C+Marc+Pamies%2C+Yishi+Xu%2C+Aitor+Gonzalez%2DAgirre%2C+and+Marta+Villegas%2E+2024%2E+FLOR%3A+On+the+effectiveness+of+language+adaptation%2E+In+Proceedings+of+the+2024+Joint)
26. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. [Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). In Proceedings of the Seventh Conferen…
27. Iria de Dios-Flores, Juan Garcia Amboage, and Marcos Garcia. 2023. Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Iria+de+Dios%2DFlores%2C+Juan+Garcia+Amboage%2C+and+Marcos+Garcia%2E+2023%2E+Dependency+resolution+at+the+syntax%2Dsemantics+interface%3A+psycholinguistic+and+computational+insights+on+control+dependencies%2E+In+Proc)
28. Pablo Gamallo, Pablo Rodríguez, Daniel Santos, Susana Sotelo, Nuno Miquelina, Silvia Paniagua, Daniela Schmidt, de Dios Flores, Iria, Paulo Quaresma, Daniel Bardanca, José Ramom Pichel, Vítor Nogueira, and Senén Barro. A Galician-Portuguese Generative Model. In Progress in Artificial Intelligence, p… [find](https://scholar.google.com/scholar?q=Pablo+Gamallo%2C+Pablo+Rodr%C3%ADguez%2C+Daniel+Santos%2C+Susana+Sotelo%2C+Nuno+Miquelina%2C+Silvia+Paniagua%2C+Daniela+Schmidt%2C+de+Dios+Flores%2C+Iria%2C+Paulo+Quaresma%2C+Daniel+Bardanca%2C+Jos%C3%A9+Ramom+Pichel%2C+V%C3%ADtor+Nogueira)
29. Iria de Dios-Flores, Carmen Magariños, Adina Ioana Vladu, John E. Ortega, José Ramom Pichel, Marcos García, Pablo Gamallo, Elisa Fernández Rei, Alberto Bugarín-Diz, Manuel González González, Senén Barro, and Xosé Luis Regueira. 2022. The nós project: Opening routes for the Galician language in the f… [find](https://scholar.google.com/scholar?q=Iria+de+Dios%2DFlores%2C+Carmen+Magari%C3%B1os%2C+Adina+Ioana+Vladu%2C+John+E%2E+Ortega%2C+Jos%C3%A9+Ramom+Pichel%2C+Marcos+Garc%C3%ADa%2C+Pablo+Gamallo%2C+Elisa+Fern%C3%A1ndez+Rei%2C+Alberto+Bugar%C3%ADn%2DDiz%2C+Manuel+Gonz%C3%A1lez+Gonz%C3%A1lez%2C+Sen%C3%A9n+Bar)
30. Pablo Gamallo, Pablo Rodríguez, Iria de Dios-Flores, Susana Sotelo, Silvia Paniagua, Daniel Bardanca, José Ramom Pichel, and Marcos Garcia. 2024. Open Generative Large Language Models for Galician. Procesamiento del Lenguaje Natural, 73:259–270. [find](https://scholar.google.com/scholar?q=Pablo+Gamallo%2C+Pablo+Rodr%C3%ADguez%2C+Iria+de+Dios%2DFlores%2C+Susana+Sotelo%2C+Silvia+Paniagua%2C+Daniel+Bardanca%2C+Jos%C3%A9+Ramom+Pichel%2C+and+Marcos+Garcia%2E+2024%2E+Open+Generative+Large+Language+Models+for+Galician%2E+Pr)
31. Daniel Deutsch and Dan Roth. 2021. Understanding the extent to which content quality metrics measure the information quality of summaries. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 300–309, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daniel+Deutsch+and+Dan+Roth%2E+2021%2E+Understanding+the+extent+to+which+content+quality+metrics+measure+the+information+quality+of+summaries%2E+In+Proceedings+of+the+25th+Conference+on+Computational+Natura)
32. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2024. Ethnologue: Languages of the World. SIL International. [find](https://scholar.google.com/scholar?q=David+M%2E+Eberhard%2C+Gary+F%2E+Simons%2C+and+Charles+D%2E+Fennig%2E+2024%2E+Ethnologue%3A+Languages+of+the+World%2E+SIL+International%2E)
33. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+Pile%3A+An+800GB+datas)
34. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
35. Marcos Garcia. 2021. Exploring the representation of word meanings in context: A case study on homonymy and synonymy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Marcos+Garcia%2E+2021%2E+Exploring+the+representation+of+word+meanings+in+context%3A+A+case+study+on+homonymy+and+synonymy%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lin)
36. Maria Giagkou, Teresa Lynn, Jane Dunne, Stelios Piperidis, and Georg Rehm. 2023. European Language Technology in 2022/2023, pages 75–94. Springer International Publishing, Cham. [find](https://scholar.google.com/scholar?q=Maria+Giagkou%2C+Teresa+Lynn%2C+Jane+Dunne%2C+Stelios+Piperidis%2C+and+Georg+Rehm%2E+2023%2E+European+Language+Technology+in+2022%2F2023%2C+pages+75%E2%80%9394%2E+Springer+International+Publishing%2C+Cham%2E)
37. Aitor Gonzalez-Agirre, Montserrat Marimon, Carlos Rodriguez-Penagos, Javier Aula-Blasco, Irene Baucells, Carme Armentano-Oller, Jorge Palomar-Giner, Baybars Kulebi, and Marta Villegas. 2024. Building a data infrastructure for a mid-resource language: The case of Catalan. In Proceedings of the 2024 J… [find](https://scholar.google.com/scholar?q=Aitor+Gonzalez%2DAgirre%2C+Montserrat+Marimon%2C+Carlos+Rodriguez%2DPenagos%2C+Javier+Aula%2DBlasco%2C+Irene+Baucells%2C+Carme+Armentano%2DOller%2C+Jorge+Palomar%2DGiner%2C+Baybars+Kulebi%2C+and+Marta+Villegas%2E+2024%2E+Building+)
38. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. [SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning](https://md.research.elamur.ai/free/en/papers/gordon-2012-semeval-2012-task-7-choice/). In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s…
39. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, W… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyat)
40. Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023. Continual pre-training of large language models: How to (re)warm your model? Preprint, arXiv:2308.04014. — [arXiv:2308.04014](https://arxiv.org/abs/2308.04014) [find](https://scholar.google.com/scholar?q=Kshitij+Gupta%2C+Benjamin+Th%C3%A9rien%2C+Adam+Ibrahim%2C+Mats+L%2E+Richter%2C+Quentin+Anthony%2C+Eugene+Belilovsky%2C+Irina+Rish%2C+and+Timoth%C3%A9e+Lesort%2E+2023%2E+Continual+pre%2Dtraining+of+large+language+models%3A+How+to+%28re%29w)
41. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pa… [find](https://scholar.google.com/scholar?q=Tahmid+Hasan%2C+Abhik+Bhattacharjee%2C+Md%2E+Saiful+Islam%2C+Kazi+Mubasshir%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+M%2E+Sohel+Rahman%2C+and+Rifat+Shahriyar%2E+2021%2E+XL%2Dsum%3A+Large%2Dscale+multilingual+abstractive+summarization)
42. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
43. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
44. Maite Heredia, Julen Etxaniz, Muitze Zulaika, Xabier Saralegi, Jeremy Barnes, and Aitor Soroa. 2024. XN-LIeu: a dataset for cross-lingual NLI in Basque. Preprint, arXiv:2404.06996. — [arXiv:2404.06996](https://arxiv.org/abs/2404.06996) [find](https://scholar.google.com/scholar?q=Maite+Heredia%2C+Julen+Etxaniz%2C+Muitze+Zulaika%2C+Xabier+Saralegi%2C+Jeremy+Barnes%2C+and+Aitor+Soroa%2E+2024%2E+XN%2DLIeu%3A+a+dataset+for+cross%2Dlingual+NLI+in+Basque%2E+Preprint%2C+arXiv%3A2404%2E06996%2E+%E2%80%94+%5BarXiv%3A2404%2E06996)
45. Tom Hosking, Phil Blunsom, and Max Bartolo. 2024. Human feedback is not gold standard. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tom+Hosking%2C+Phil+Blunsom%2C+and+Max+Bartolo%2E+2024%2E+Human+feedback+is+not+gold+standard%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
46. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
47. Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023. [Evaluating open-domain question answering in the era of large language models](https://md.research.elamur.ai/free/en/papers/kamalloo-2023-evaluating-open-domain-question-answering/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5591–5606, Toronto, Canada. A…
48. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). Preprint, arXiv:2405.01535.
49. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). NeurIPS.
50. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loub… — [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
51. Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. 2023. Holistic evaluation of text-… [find](https://scholar.google.com/scholar?q=Tony+Lee%2C+Michihiro+Yasunaga%2C+Chenlin+Meng%2C+Yifan+Mai%2C+Joon+Sung+Park%2C+Agrim+Gupta%2C+Yunzhi+Zhang%2C+Deepak+Narayanan%2C+Hannah+Benita+Teufel%2C+Marco+Bellagente%2C+Minguk+Kang%2C+Taesung+Park%2C+Jure+Leskovec%2C+Ju)
52. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
53. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
54. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
55. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+R%2E+Bowman%2C+and+Shi+Feng%2E+2024%2E+LLM+evaluators+recognize+and+favor+their+own+generations%2E+Preprint%2C+arXiv%3A2404%2E13076%2E)
56. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. [The lambada dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). Preprint, arXiv:1606.06031.
57. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
58. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. Preprint, arXiv:2306.0111… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+RefinedWeb+dataset+for+F)
59. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese Large Language Models, page 226–240. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Ramon+Pires%2C+Hugo+Abonizio%2C+Thales+Sales+Almeida%2C+and+Rodrigo+Nogueira%2E+2023%2E+Sabi%C3%A1%3A+Portuguese+Large+Language+Models%2C+page+226%E2%80%93240%2E+Springer+Nature+Switzerland%2E)
60. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)
61. Georg Rehm and Andy Way. 2023. European Language Equality: A Strategic Agenda for Digital Language Equality. Springer Cham. [find](https://scholar.google.com/scholar?q=Georg+Rehm+and+Andy+Way%2E+2023%2E+European+Language+Equality%3A+A+Strategic+Agenda+for+Digital+Language+Equality%2E+Springer+Cham%2E)
62. Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinmeng Ji, Tingting Cui, Tao Liu, Jinwang Song, Hongying Zan, Sun Li, and Deyi Xiong. 2024. OpenEval: Benchmarking chinese LLMs across capability, alignment and safety. Preprint, arXiv:2403.12316. [find](https://scholar.google.com/scholar?q=Chuang+Liu%2C+Linhao+Yu%2C+Jiaxuan+Li%2C+Renren+Jin%2C+Yufei+Huang%2C+Ling+Shi%2C+Junhui+Zhang%2C+Xinmeng+Ji%2C+Tingting+Cui%2C+Tao+Liu%2C+Jinwang+Song%2C+Hongying+Zan%2C+Sun+Li%2C+and+Deyi+Xiong%2E+2024%2E+OpenEval%3A+Benchmarking+)
63. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
64. Ricardo Lopes, Joao Magalhaes, and David Semedo. 2024. GlórIA: A generative and open large language model for Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, pages 441–453, Santiago de Compostela, Galicia/Spain. Association for Comp… [find](https://scholar.google.com/scholar?q=Ricardo+Lopes%2C+Joao+Magalhaes%2C+and+David+Semedo%2E+2024%2E+Gl%C3%B3rIA%3A+A+generative+and+open+large+language+model+for+Portuguese%2E+In+Proceedings+of+the+16th+International+Conference+on+Computational+Processin)
65. Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for NLG systems. ACM Comput. Surv., 55(2). [find](https://scholar.google.com/scholar?q=Ananya+B%2E+Sai%2C+Akash+Kumar+Mohankumar%2C+and+Mitesh+M%2E+Khapra%2E+2022%2E+A+survey+of+evaluation+metrics+used+for+NLG+systems%2E+ACM+Comput%2E+Surv%2E%2C+55%282%29%2E)
66. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natur… [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Ondr%CC%8Cej+Du%C5%A1ek%2C+Amanda+Cercas+Curry%2C+and+Verena+Rieser%2E+2017%2E+Why+we+need+Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2019%2E+Social+IQa%3A+Commonsense+reas)
67. Teven Le Scao et al. 2023. BLOOM: A 176B-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao+et+al%2E+2023%2E+BLOOM%3A+A+176B%2Dparameter+open%2Daccess+multilingual+language+model%2E+Preprint%2C+arXiv%3A2211%2E05100%2E)
68. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. Preprint, arXiv:2310.11324. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2023%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+I+learned+to+start+worrying+about+prompt+formatt)
69. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
70. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). In The Eleventh International Conference on Learning Representa…
71. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2023. [mGPT: Few-shot learners go multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/). Preprint, arXiv:2204.07580.
72. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. IndicGen-Bench: A multilingual benchmark to evaluate generation capabilities of LLMs on Indic languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Harman+Singh%2C+Nitish+Gupta%2C+Shikhar+Bharadwaj%2C+Dinesh+Tewari%2C+and+Partha+Talukdar%2E+2024%2E+IndicGen%2DBench%3A+A+multilingual+benchmark+to+evaluate+generation+capabilities+of+LLMs+on+Indic+languages%2E+In+Pro)
73. Aarohi Srivastava et al. 2023. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Preprint, arXiv:2206.04615.
74. Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8640–8665, Toront… [find](https://scholar.google.com/scholar?q=Yiming+Wang%2C+Zhuosheng+Zhang%2C+and+Rui+Wang%2E+2023%2E+Element%2Daware+summarization+with+large+language+models%3A+Expert%2Daligned+evaluation+and+chain%2Dof%2Dthought+method%2E+In+Proceedings+of+the+61st+Annual+Meeti)
75. Jason Wei, Maarten Paul Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Mingbo Dai, and Quoc V. Le. 2022a. Fine-tuned language models are zero-shot learners. In Proceedings of the Tenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Paul+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+Mingbo+Dai%2C+and+Quoc+V%2E+Le%2E+2022a%2E+Fine%2Dtuned+language+models+are+zero%2Dshot+learners%2E+In+Proceedings)
76. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on M…
77. Minghao Wu and Alham Fikri Aji. 2023. [Style over substance: Evaluation biases for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/). Preprint, arXiv:2307.03025.
78. Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… [find](https://scholar.google.com/scholar?q=Vikas+Yadav%2C+Steven+Bethard%2C+and+Mihai+Surdeanu%2E+2019%2E+Quick+and+%28not+so%29+dirty%3A+Unsupervised+selection+of+justification+sentences+for+multi%2Dhop+question+answering%2E+In+Proceedings+of+the+2019+Conferen)
79. Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2022. BasqueGLUE: A natural language understanding benchmark for Basque. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1603– 1612, Marseille, France. European Language Resources… [find](https://scholar.google.com/scholar?q=Gorka+Urbizu%2C+I%C3%B1aki+San+Vicente%2C+Xabier+Saralegi%2C+Rodrigo+Agerri%2C+and+Aitor+Soroa%2E+2022%2E+BasqueGLUE%3A+A+natural+language+understanding+benchmark+for+Basque%2E+In+Proceedings+of+the+Thirteenth+Language+Re)
80. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [PAWS-X: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…
81. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [SuperGLUE: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
82. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024. [FLASK: Fine-grained language model evaluation based on alignment skill sets](https://md.research.elamur.ai/free/en/papers/ye-2023-flask-fine-grained-language-model/). In The Twelfth International Conference on Learning Representations.
83. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E)
84. GLUE: Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. Preprint, arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=GLUE%3A+Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+Preprint%2C+arXiv%3A1904%2E09675%2E)
85. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. [Benchmarking Large Language Models for News Summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Transactions of the Association for Computational Linguistics, 12:39–57.
86. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. [PAWS: Paraphrase adversaries from word scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, M…
87. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume 3…
88. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. [Aya model: An instruction fine…](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)

*Updated 1 November 2026*