---
title: "Evaluating Small Language Models for — Tamiosso 2026"
description: "Small language models (SLMs) are cheaper to run than large proprietary systems."
authors: ["Gustavo Lopes Tamiosso", "Rafael Oleques Nunes", "Dennis Giovani Balreira"]
year: 2026
venue: "Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1"
source: "https://aclanthology.org/2026.propor-1.94/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/tamiosso-2026-evaluating-small-language-models-english/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/tamiosso-2026-evaluating-small-language-models-english/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/2/) / Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and Quantization

# Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and Quantization

**Authors:** Gustavo Lopes Tamiosso, [Rafael Oleques Nunes](https://md.research.elamur.ai/free/authors/rafael-oleques-nunes/), [Dennis Giovani Balreira](https://md.research.elamur.ai/free/authors/dennis-giovani-balreira/) · **Year:** 2026 · **Venue:** Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.propor-1.94/) · [PDF](https://aclanthology.org/2026.propor-1.94.pdf)

## Summary
Small language models (SLMs) are cheaper to run than large proprietary systems. However, there has been little systematic evidence on how well they translate English into Portuguese, and on how model size and low-bit compression affect that quality. The authors benchmark dozens of SLMs from 135M to 20B parameters, drawn from the Gemma, Qwen, LLaMA, Mistral, OLMo, DeepSeek, SmolLM2 and GPT families. Open-weight models are run in FP16, Q8_0 and Q4_K_M formats with one fixed zero-shot instruction prompt. Outputs are scored with BLEU, chrF and BERTScore on FLORES-101 and OPUS-100. Sentence-level differences are tested non-parametrically, and their practical size is measured with Cliff's δ. The main findings are:
- **8-bit compression:** essentially lossless.
- **4-bit compression:** sometimes statistically detectable, but rarely meaningful, and mostly harmful only for the smallest models.
- **Model size:** a weak predictor of quality.
- **Best open model:** a 4B Gemma 3n comes close to GPT-4o.

These results matter for anyone building Portuguese translation pipelines on local, consumer-grade hardware.

## Key points
- **Data:** FLORES-101 Portuguese subset (por_Latn, 1,012 test sentences, formal text) and 9,271 English–Portuguese pairs sampled from the OPUS-100 test split (mixed domains such as subtitles, parliament and software).
- **Statistics:** Friedman tests over paired sentence scores, Wilcoxon signed-rank post-hoc comparisons with Holm correction, Shapiro–Wilk normality checks, and paired Cliff's δ for effect size.
- **Effect of Q4_K_M:** significant in 6 of 15 configurations on FLORES-101 and 9 of 15 on OPUS-100, with mostly negligible effects. LLaMA 3.2 (1B) was the sole model where the degradation was reliably present, though still modest in size (δ ≈ 0.18–0.23). Gemma 3n stayed stable across formats, which the authors attribute to its MatFormer/PLE design.
- **Effect of scale:** size correlates only weakly with BERTScore (Pearson r ≈ 0.3). Large effects appear only across extreme size gaps, for example SmolLM2 135M vs 1.7B (δ = 0.99) and Qwen 600M vs 4B (δ = 0.96).
- **GPT-4o vs Gemma 3n 4B FP16:**
  - On FLORES-101, GPT-4o wins 68.8% of sentences (δ = 0.39, medium).
  - On OPUS-100, it wins only 53.5% (δ = 0.14, negligible).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tamiosso2026evaluating,
  title = {{Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and Quantization}},
  author = {Tamiosso, Gustavo Lopes and Nunes, Rafael Oleques and Balreira, Dennis Giovani},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.94/}
}
```

- [← previous Evaluating Native-Speaker Preferences on Machine Translation and Post-Edits for Five African Languages](https://md.research.elamur.ai/free/papers/oirghi-2026-evaluating-native-speaker-preferences-machine/)
- [next → Evaluating the Impact of Source Diversity for RAG in Historical Research](https://md.research.elamur.ai/free/papers/mahadeshwar-2026-evaluating-impact-source-diversity-rag/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/2/)

## References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Maximiliana Behnke, Nikolay Bogoychev, Alham Fikri Aji, Kenneth Heafield, Graeme Nail, Qianqian Zhu, Svetlana Tchistiakova, Jelmer van der Linde, Pinzhen Chen, Sidharth Kashyap, and Roman Grundkiewicz. 2021. Efficient machine translation with model pruning and quantization. In Proceedings of the Sixth Conference on Machine Translation, pages 775–780, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maximiliana+Behnke%2C+Nikolay+Bogoychev%2C+Alham+Fikri+Aji%2C+Kenneth+Heafield%2C+Graeme+Nail%2C+Qianqian+Zhu%2C+Svetlana+Tchistiakova%2C+Jelmer+van+der+Linde%2C+Pinzhen+Chen%2C+Sidharth+Kashyap%2C+and+Roman+Grundkiewicz)
2. Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114(3):494–509. [↗](https://doi.org/10.1037//0033-2909.114.3.494)
3. Gemma Team. 2025. Gemma 3n. Accessed November 2025. [find](https://scholar.google.com/scholar?q=Gemma+Team%2E+2025%2E+Gemma+3n%2E+Accessed+November+2025%2E)
4. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. [↗](https://doi.org/10.1162/tacl_a_00474)
5. Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12186–12215, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Renren+Jin%2C+Jiangcun+Du%2C+Wuwei+Huang%2C+Wei+Liu%2C+Jian+Luan%2C+Bin+Wang%2C+and+Deyi+Xiong%2E+2024%2E+A+comprehensive+evaluation+of+quantization+strategies+for+large+language+models%2E+In+Findings+of+the+Associatio)
6. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
7. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w15-3049) [our card](https://md.research.elamur.ai/free/papers/popovic-2015-chrf-character-n-gram-f/)
8. Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. 2020. Fully quantized transformer for machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1–14, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gabriele+Prato%2C+Ella+Charlaix%2C+and+Mehdi+Rezagholizadeh%2E+2020%2E+Fully+quantized+transformer+for+machine+translation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+1%E2%80%9314)
9. Jerry Quinn and Miguel Ballesteros. 2018. Pieces of eight: 8-bit neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 114–120, New Orleans -Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jerry+Quinn+and+Miguel+Ballesteros%2E+2018%2E+Pieces+of+eight%3A+8%2Dbit+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Li)
10. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems, pages 403– 417, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=F%C3%A1bio+Souza%2C+Rodrigo+Nogueira%2C+and+Roberto+Lotufo%2E+2020%2E+Bertimbau%3A+Pretrained+bert+models+for+brazilian+portuguese%2E+In+Intelligent+Systems%2C+pages+403%E2%80%93+417%2C+Cham%2E+Springer+International+Publishing%2E)
11. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2214%E2%80%932218%2C+Istanbul%2C+Turk)
12. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. [↗](https://arxiv.org/abs/2505.09388) [our card](https://md.research.elamur.ai/free/papers/yang-2025-qwen3-technical-report/)
13. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628– 1639, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2004.11867) [our card](https://md.research.elamur.ai/free/papers/zhang-2020-improving-massively-multilingual-neural-machine/)
14. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1904.09675)