---
title: "Efficient Fine-Tuning Methods for Portuguese Question"
description: "They test 40 configurations that cross LoRA, DoRA, QLoRA and QDoRA with model sizes and learning rates… 9.56 F1 on Base. 33 references."
authors: ["Mariela M. Nina", "Caio Veloso Costa", "Lilian Berton", "Didier A. Vega-Oliveros"]
year: 2026
venue: "Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1"
source: "https://aclanthology.org/2026.propor-1.91/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nina-2026-efficient-fine-tuning-methods-portuguese/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nina-2026-efficient-fine-tuning-methods-portuguese/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/) / Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

# Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

**Authors:** Mariela M. Nina, Caio Veloso Costa, [Lilian Berton](https://md.research.elamur.ai/free/en/authors/lilian-berton/) and Didier A. Vega-Oliveros · **Year:** 2026 · **Venue:** [Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1](https://md.research.elamur.ai/free/en/venues/propor/) ([International Conference on Computational Processing of Portuguese 2026](https://md.research.elamur.ai/free/en/venues/propor/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.propor-1.91/) · [PDF](https://aclanthology.org/2026.propor-1.91.pdf) · [OpenAlex](https://openalex.org/W7140236749)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

The paper asks whether parameter-efficient adaptation and low-bit quantization, mostly validated on English, also work for extractive question answering in Brazilian Portuguese, where compute budgets are often limited. The authors fine-tune BERTimbau (Base, 110M; Large, 335M) on SQuAD-BR, a Portuguese translation of SQuAD v1. They test 40 configurations that cross LoRA, DoRA, QLoRA and QDoRA with model sizes and learning rates, using full fine-tuning as the reference. They score accuracy (F1, Exact Match), training time and peak GPU memory.

LoRA on the Large model gets close to the full fine-tuning baseline while cutting training time by roughly three quarters. PEFT needs much higher learning rates than usual, and 4-bit quantization hurts the larger model about half as much as the smaller one. A smaller exploratory test of the generative models Tucano and Sabiá shows they can reach similar F1 with LoRA but use far more resources.

The work gives practical, Green-AI-oriented guidance: compact encoders adapted with PEFT are a cheap, strong option for Portuguese extractive QA.

## Key points

- **LoRA on BERTimbau-Large** reached F1 = 81.32 against 84.86 for full fine-tuning, which is 95.8% of the baseline, with 73.5% less training time. DoRA matched LoRA but added about 28% training time, so it brought no practical benefit.
- **Learning rate:** raising it to 2e-4 improved PEFT runs by up to +19.71 F1. At the same rate, full fine-tuning of the Large model collapsed to F1 = 3.02. The authors read this as low-rank updates acting like implicit regularization.
- **QLoRA and model size:** compared with the baseline, QLoRA lost 9.56 F1 on Base but only 4.83 on Large, where it kept 94.3% of baseline performance. Peak memory fell by 86.9% on the Base model, to 1,897 MB, and by 81.9% on the Large model, to 3,281 MB, putting both within reach of consumer GPUs.
- **Generative models:** Sabiá-7B with LoRA scored F1 = 78.10, roughly level with BERTimbau-Base at 78.01. It needed 4.2× the GPU memory (15,642 vs 3,687 MB) and about 3× the training time. BERTimbau-Large with LoRA outperformed every generative model tested, including Tucano.
- **Recommendation:** prefer BERTimbau-Large with LoRA or QLoRA at high learning rates. Treat full fine-tuning only as a baseline, and avoid aggressive quantization of Base models unless memory is very tight.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Universidade Federal de São Paulo](https://ror.org/02k5swt12).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Mariela M. Nina, Caio Veloso Costa, Lilian Berton and Didier A. Vega-Oliveros. 2026. Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nina2026efficient,
  title = {{Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs}},
  author = {Mariela M. Nina, Caio Veloso Costa, Lilian Berton and Didier A. Vega-Oliveros},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.91/}
}
```

- [← previous Efficient Agent Evaluation via Diversity-Guided User Simulation](https://md.research.elamur.ai/free/en/papers/nakash-2026-efficient-agent-evaluation-diversity-guided/)
- [next → Efficient KL Divergence Estimation via Truncated Top-K Integration for Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2026-efficient-kl-divergence-estimation-truncated/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/)

## References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). Preprint, arXiv:2012.13255.
2. Gian Franco Condori-Luna, Didier Vega-Oliveros, and Marcelo da Silva Reis. 2026. Reducing dependence on labeled data: A self-supervised fine-tuning approach for low-resource language models. In Intelligent Systems, pages 395–409, Cham. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Gian+Franco+Condori%2DLuna%2C+Didier+Vega%2DOliveros%2C+and+Marcelo+da+Silva+Reis%2E+2026%2E+Reducing+dependence+on+labeled+data%3A+A+self%2Dsupervised+fine%2Dtuning+approach+for+low%2Dresource+language+models%2E+In+Intell)
3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
4. Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2025. Tucano: Advancing neural text generation for portuguese. Patterns, 6(11):101325. [find](https://scholar.google.com/scholar?q=Nicholas+Kluge+Corr%C3%AAa%2C+Aniket+Sen%2C+Sophia+Falk%2C+and+Shiza+Fatimah%2E+2025%2E+Tucano%3A+Advancing+neural+text+generation+for+portuguese%2E+Patterns%2C+6%2811%29%3A101325%2E)
5. E. da Silva, J. Laterza, and T. Faleiros. 2022. New state-of-the-art for question answering on portuguese squad v1.1. In Anais do X Symposium on Knowledge Discovery, Mining and Learning, pages 98–105, Porto Alegre, RS, Brasil. SBC. [find](https://scholar.google.com/scholar?q=E%2E+da+Silva%2C+J%2E+Laterza%2C+and+T%2E+Faleiros%2E+2022%2E+New+state%2Dof%2Dthe%2Dart+for+question+answering+on+portuguese+squad+v1%2E1%2E+In+Anais+do+X+Symposium+on+Knowledge+Discovery%2C+Mining+and+Learning%2C+pages+98%E2%80%93105%2C)
6. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. [8-bit optimizers via block-wise quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/). Preprint, arXiv:2110.02861.
7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). Preprint, arXiv:2305.14314.
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). Preprint, arXiv:1810.04805.
9. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. Preprint, arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2023%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+Preprint%2C+arXiv%3A2210%2E17323%2E)
10. Christian Freitas, Didier A. Vega-Oliveros, and Lilian Berton. 2025. Enhancing industrial data access with text-to-sql using portuguese llms and langgraph. In 2025 12th International Conference on Soft Computing & Machine Intelligence (ISCMI), pages 278–282. [find](https://scholar.google.com/scholar?q=Christian+Freitas%2C+Didier+A%2E+Vega%2DOliveros%2C+and+Lilian+Berton%2E+2025%2E+Enhancing+industrial+data+access+with+text%2Dto%2Dsql+using+portuguese+llms+and+langgraph%2E+In+2025+12th+International+Conference+on+Sof)
11. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). Preprint, arXiv:2110.04366.
12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
13. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. Preprint, arXiv:1902.00751. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+Pre)
14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). Preprint, arXiv:2106.09685.
15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). Preprint, arXiv:2001.08361.
16. Diederik P. Kingma and Jimmy Ba. 2017. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). Preprint, arXiv:1412.6980.
17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). Preprint, arXiv:2104.08691.
18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Preprint, arXiv:2101.00190. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+Preprint%2C+arXiv%3A2101%2E00190%2E)
19. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. Preprint, arXiv:2402.09353. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+Preprint%2C+arXiv%3A2402%2E09353%2E)
20. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+Preprint%2C+arXiv%3A1711%2E05101%2E)
21. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft). [↗](https://github.com/huggingface/peft)
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). Preprint, arXiv:1606.05250.
23. Tim Salimans and Diederik P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Preprint, arXiv:1602.07868. [find](https://scholar.google.com/scholar?q=Tim+Salimans+and+Diederik+P%2E+Kingma%2E+2016%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+Preprint%2C+arXiv%3A1602%2E07868%2E)
24. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems, pages 403– 417, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=F%C3%A1bio+Souza%2C+Rodrigo+Nogueira%2C+and+Roberto+Lotufo%2E+2020%2E+Bertimbau%3A+Pretrained+bert+models+for+brazilian+portuguese%2E+In+Intelligent+Systems%2C+pages+403%E2%80%93+417%2C+Cham%2E+Springer+International+Publishing%2E)
25. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. [Energy and policy considerations for deep learning in nlp](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). Preprint, arXiv:1906.02243.
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Preprint, arXiv:1706.03762.
27. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Preprint, arXiv:1905.00537.
28. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). Preprint, arXiv:1804.07461.
29. Jing Yang, Didier Vega-Oliveros, Taís Seibt, and Anderson Rocha. 2022. Explainable fact-checking through question answering. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8952–8956. [find](https://scholar.google.com/scholar?q=Jing+Yang%2C+Didier+Vega%2DOliveros%2C+Ta%C3%ADs+Seibt%2C+and+Anderson+Rocha%2E+2022%2E+Explainable+fact%2Dchecking+through+question+answering%2E+In+ICASSP+2022+%2D+2022+IEEE+International+Conference+on+Acoustics%2C+Speech+an)
30. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. [AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/). In International Conference on Learning Representations (ICLR).
31. Ricardo Mazza Zago and Luciane Agnoletti dos Santos Pedotti. 2024. Bertugues: A novel bert transformer model pre-trained for brazilian portuguese. Semina: Ciências Exatas e Tecnológicas, 45:e50630. [find](https://scholar.google.com/scholar?q=Ricardo+Mazza+Zago+and+Luciane+Agnoletti+dos+Santos+Pedotti%2E+2024%2E+Bertugues%3A+A+novel+bert+transformer+model+pre%2Dtrained+for+brazilian+portuguese%2E+Semina%3A+Ci%C3%AAncias+Exatas+e+Tecnol%C3%B3gicas%2C+45%3Ae50630%2E)
32. Mariela M. Nina and Didier A. Vega-Oliveros. 2025. Maximizing model adaptation for low-resource languages: A progressive unfreezing strategy for spanish-aymara translation. In 2025 12th International Conference on Soft Computing & Machine Intelligence (ISCMI), pages 259–263. [find](https://scholar.google.com/scholar?q=Mariela+M%2E+Nina+and+Didier+A%2E+Vega%2DOliveros%2E+2025%2E+Maximizing+model+adaptation+for+low%2Dresource+languages%3A+A+progressive+unfreezing+strategy+for+spanish%2Daymara+translation%2E+In+2025+12th+International+)
33. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese Large Language Models, page 226–240. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Ramon+Pires%2C+Hugo+Abonizio%2C+Thales+Sales+Almeida%2C+and+Rodrigo+Nogueira%2E+2023%2E+Sabi%C3%A1%3A+Portuguese+Large+Language+Models%2C+page+226%E2%80%93240%2E+Springer+Nature+Switzerland%2E)

*Updated 1 November 2026*