--- title: "Benchmarking Retrieval-Augmented Generation for Scientific" description: "The paper asks whether retrieval-augmented generation helps small open language models answer scientific multiple-choice… 1 reference." authors: ["José Matos", "Catarina Silva", "Hugo Gonçalo Oliveira"] year: 2026 venue: "Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026" source: "https://aclanthology.org/2026.nslp-1.3/" doi: "10.63317/3muergicuxwk" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/matos-2026-benchmarking-retrieval-augmented-generation-scientific/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/matos-2026-benchmarking-retrieval-augmented-generation-scientific/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese # Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese **Authors:** José Matos, [Catarina Silva](https://md.research.elamur.ai/free/en/authors/catarina-silva/) and [Hugo Gonçalo Oliveira](https://md.research.elamur.ai/free/en/authors/hugo-goncalo-oliveira/) · **Year:** 2026 · **Venue:** [Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/nslp/) ([Natural Scientific Language Processing (NSLP) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/nslp/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.nslp-1.3/) · [PDF](https://aclanthology.org/2026.nslp-1.3.pdf) · [DOI 10.63317/3muergicuxwk](https://doi.org/10.63317/3muergicuxwk) · [OpenAlex](https://openalex.org/W7166546099) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary The paper asks whether retrieval-augmented generation helps small open language models answer scientific multiple-choice questions in European Portuguese (pt-PT), and how much the outcome depends on retrieval design. Earlier Portuguese RAG studies focused mainly on the Brazilian variety. The authors evaluate five instruction-tuned models (4–12B) using the Portuguese portion of the Global MMLU Lite test data. External evidence comes from a vector index built from more than 32,000 Portuguese scientific documents in the CorEGe-PT corpus. Each model runs closed-book and in 16 retrieval setups, which vary the dense encoder (multilingual or Portuguese-specific), the use of a reranker, and the number of retrieved chunks. The benefit of retrieval varies strongly by model: - **Gemma3:** large and consistent gains. - **AMALIA and EuroLLM:** small gains that depend on the configuration. - **Qwen3-8B:** the strongest closed-book model, but slightly worse with retrieval, especially with longer contexts. The study gives a reproducible pt-PT baseline. It shows that retrieval settings need tuning per model and that retriever and reranker language and domain should match the task. ## Key points - **Setup:** - Benchmark: Global MMLU Lite PT split (400 professionally translated items). - Models: AMALIA, EuroLLM-9B-Instruct, Qwen3-8B, Gemma3-4B-IT, Gemma3-12B-IT. - Evaluation: 3-shot prompting with log-likelihood scoring in lm_evaluation_harness. - **Retrieval pipeline:** - Chunking: section-based splits, then 500-character chunks with 50-character overlap, stored in ChromaDB. - Encoders: paraphrase-multilingual-mpnet-base-v2 vs. serafim-335m-portuguese-pt-sentence-encoder-ir. - Reranker: gte-multilingual-reranker-base. - Context size: k ∈ {1, 3, 5, 10}. - **Largest significant gains (p \< 0.05):** Gemma-3-12B-IT went from 55.5% to 71.0% (+15.5 pp), and Gemma-3-4B-IT from 50.7% to 60.5% (+9.8 pp). - **Small or negative effects:** AMALIA improved by at most +2.7 pp and EuroLLM by at most +2.2 pp, with mean gains near zero. Qwen3-8B (73.0% closed-book) lost 2.89 pp on average. - **Reranking:** it did not reliably raise accuracy, but it tended to move the best k toward smaller contexts. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Coimbra](https://md.research.elamur.ai/free/en/universities/university-of-coimbra/) — University, Coimbra, Portugal · 2 works 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works 🧭 Topics (2): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works ## Cite Cite as: José Matos, Catarina Silva and Hugo Gonçalo Oliveira. 2026. Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese. Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{matos2026benchmarking, title = {{Benchmarking Retrieval-Augmented Generation for Scientific Knowledge QA in European Portuguese}}, author = {José Matos, Catarina Silva and Hugo Gonçalo Oliveira}, booktitle = {Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026}, year = {2026}, doi = {10.63317/3muergicuxwk}, url = {https://doi.org/10.63317/3muergicuxwk} } ``` - [← previous Benchmarking Portuguese Open Information Extraction](https://md.research.elamur.ai/free/en/papers/silva-2026-benchmarking-portuguese-open-information-extraction/) - [next → Benchmarking Temporal Reasoning and Alignment Across Chinese Dynasties](https://md.research.elamur.ai/free/en/papers/wang-2026-benchmarking-temporal-reasoning-alignment-across/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Leandro Costa and João Baptista de Oliveira e Souza-Filho. 2024. Adapting LLMs to New Domains: A Comparative Study of Fine-Tuning and RAG strategies for Portuguese QA Tasks. In Proceedings of the 15th Brazilian Symposium in Information and Human Language Technology, pages 217–227, Belém do Pará, Bra… [find](https://scholar.google.com/scholar?q=Leandro+Costa+and+Jo%C3%A3o+Baptista+de+Oliveira+e+Souza%2DFilho%2E+2024%2E+Adapting+LLMs+to+New+Domains%3A+A+Comparative+Study+of+Fine%2DTuning+and+RAG+strategies+for+Portuguese+QA+Tasks%2E+In+Proceedings+of+the+15th) *Updated 1 November 2026*