---
title: "Using machine learning and large language models for the"
description: "The authors therefore offer a first systematic comparison of scoring approaches for this exam. 27 references."
authors: ["Rafael Oleques Nunes", "Bernardo Cobalchini Zietolie", "Ricardo Zanini De Costa", "Rodrigo Brock da Silva", "João Victor Piardi Pacheco", "Rafaela Dall'Agnol da Rocha", "Dennis Giovani Balreira", "Elisa Marchioro Stumpf", "Juliana Roquele Schoffen"]
year: 2026
venue: "Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1"
source: "https://aclanthology.org/2026.propor-1.85/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
title_orig: "Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras"
title_lang: "pt"
canonical: "https://research.elamur.ai/free/en/papers/nunes-2026-uso-de-tecnicas-de-aprendizado/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nunes-2026-uso-de-tecnicas-de-aprendizado/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/) / Using machine learning and large language models for the automatic assessment of Celpe-Bras exam texts

# Using machine learning and large language models for the automatic assessment of Celpe-Bras exam texts

*Original (Portuguese): Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras*

**Authors:** [Rafael Oleques Nunes](https://md.research.elamur.ai/free/en/authors/rafael-oleques-nunes/), Bernardo Cobalchini Zietolie, Ricardo Zanini De Costa, Rodrigo Brock da Silva, João Victor Piardi Pacheco, Rafaela Dall'Agnol da Rocha, [Dennis Giovani Balreira](https://md.research.elamur.ai/free/en/authors/dennis-giovani-balreira/), [Elisa Marchioro Stumpf](https://md.research.elamur.ai/free/en/authors/elisa-marchioro-stumpf/) and Juliana Roquele Schoffen · **Year:** 2026 · **Venue:** [Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1](https://md.research.elamur.ai/free/en/venues/propor/) ([International Conference on Computational Processing of Portuguese 2026](https://md.research.elamur.ai/free/en/venues/propor/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.propor-1.85/) · [PDF](https://aclanthology.org/2026.propor-1.85.pdf)

## Summary
The paper tackles automated essay scoring (AES) for the written part of Celpe-Bras, Brazil's official proficiency exam in Portuguese as an additional language. Teachers preparing candidates face heavy grading loads, and learners have few affordable resources aligned with the exam's construct. The authors therefore offer a first systematic comparison of scoring approaches for this exam. They used texts from the Corpus Celpe-Bras (CorCel) and framed scoring as six-class classification over grades 0–5. Three model families were compared:

- **Classical baselines:** Naive Bayes, SVM and Random Forest on TF-IDF features.
- **Portuguese encoder models:** BERT variants.
- **Encoder-decoder models:** mBART-50 and a Portuguese T5.

The BERT-based models, especially BERTimbau (large), scored best and reached moderate agreement with official raters, though at a much higher computational cost. SVM came fairly close for a fraction of the compute. The work shows that automated scoring of Celpe-Bras texts is feasible. It also sets a baseline for future feedback tools for learners of Portuguese.

## Key points
- **Data:** CorCel holds 15,315 rated texts (about 3M words) from four exam editions (2015-2 to 2017-1). The experiments used one task's subset of 2,293 texts with integer grades, which is heavily imbalanced (only 25 texts scored 0; most scored 2–3).
- **Models:**
  - Naive Bayes, SVM and RF, with stopword removal and snowball stemming.
  - BERTimbau (base and large), Albertina (100m) and BERTugues (base).
  - mBART-50 and ptt5-v2-base.
- **Evaluation:** stratified 5-fold cross-validation, with 10% of each training split held out for validation. Metrics were reported in macro and weighted forms, and models were trained on Colab T4 GPUs.
- **Best results:** BERTimbau (large) on the optimized data reached accuracy 0.4128 and F1 0.4049. BERT variants achieved QWK of about 0.59 on the full dataset and about 0.57 on the optimized one.
- **Weaker models:**
  - Albertina over-predicted grades 2 and 3 and struggled with minority classes.
  - Naive Bayes, RF and mBART-50 performed poorly.
  - SVM and T5 performed similarly to each other.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Rafael Oleques Nunes, Bernardo Cobalchini Zietolie, Ricardo Zanini De Costa, Rodrigo Brock da Silva, João Victor Piardi Pacheco, Rafaela Dall'Agnol da Rocha, Dennis Giovani Balreira, Elisa Marchioro Stumpf and Juliana Roquele Schoffen. 2026. Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nunes2026uso,
  title = {{Uso de técnicas de Aprendizado de Máquina e Modelos de Língua de Larga Escala para avaliação automática de textos do exame Celpe-Bras}},
  author = {Rafael Oleques Nunes, Bernardo Cobalchini Zietolie, Ricardo Zanini {De Costa}, Rodrigo Brock {da Silva}, João Victor Piardi Pacheco, Rafaela Dall'Agnol {da Rocha}, Dennis Giovani Balreira, Elisa Marchioro Stumpf and Juliana Roquele Schoffen},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.85/}
}
```

- [← previous Using LLMs for item creation: Validating the potential of automatically generated sentence repetition test items for language assessment](https://md.research.elamur.ai/free/en/papers/lober-2026-llms-item-creation-validating-potential/)
- [next → Using Temporal Stability to Probe the Syllabification of Medial Geminates: Evidence from Moroccan Arabic](https://md.research.elamur.ai/free/en/papers/nirheche-2026-temporal-stability-probe-syllabification-medial/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/55/)

## References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yigal Attali and Jill Burstein. 2004. Automated essay scoring with e-rater® v. 2.0. ETS Research Report Series, 2004(2):i–21. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2004%2E+Automated+essay+scoring+with+e%2Drater%C2%AE+v%2E+2%2E0%2E+ETS+Research+Report+Series%2C+2004%282%29%3Ai%E2%80%9321%2E)
2. Claire Barale, Michael Rovatsos, and Nehal Bhuta. 2023. Do language models learn about legal entity types during pretraining? In Proceedings of the Natural Legal Language Processing Workshop, volume 2023, pages 25–37. [find](https://scholar.google.com/scholar?q=Claire+Barale%2C+Michael+Rovatsos%2C+and+Nehal+Bhuta%2E+2023%2E+Do+language+models+learn+about+legal+entity+types+during+pretraining%3F+In+Proceedings+of+the+Natural+Legal+Language+Processing+Workshop%2C+volume+2)
3. Jing Chen, James H Fife, Isaac I Bejar, and André A Rupp. 2016. Building e-rater® scoring models using machine learning methods. ETS Research Report Series, 2016(1):1–12. [find](https://scholar.google.com/scholar?q=Jing+Chen%2C+James+H+Fife%2C+Isaac+I+Bejar%2C+and+Andr%C3%A9+A+Rupp%2E+2016%2E+Building+e%2Drater%C2%AE+scoring+models+using+machine+learning+methods%2E+ETS+Research+Report+Series%2C+2016%281%29%3A1%E2%80%9312%2E)
4. Tobias Deußer, Cong Zhao, Lorenz Sparrenberg, Daniel Uedelhoven, Armin Berger, Maren Pielka, Lars Hillebrand, Christian Bauckhage, and Rafet Sifa. 2024. A comparative study of large language models for named entity recognition in the legal domain. In 2024 IEEE International Conference on Big Data (B… [find](https://scholar.google.com/scholar?q=Tobias+Deu%C3%9Fer%2C+Cong+Zhao%2C+Lorenz+Sparrenberg%2C+Daniel+Uedelhoven%2C+Armin+Berger%2C+Maren+Pielka%2C+Lars+Hillebrand%2C+Christian+Bauckhage%2C+and+Rafet+Sifa%2E+2024%2E+A+comparative+study+of+large+language+models+fo)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
6. Luiza Sarmento Divino. 2024. Contribuições da linguística de corpus para a definição de níveis de proficiência escrita no exame celpe-bras. Master’s thesis, Universidade Federal do Rio Grande do Sul. [find](https://scholar.google.com/scholar?q=Luiza+Sarmento+Divino%2E+2024%2E+Contribui%C3%A7%C3%B5es+da+lingu%C3%ADstica+de+corpus+para+a+defini%C3%A7%C3%A3o+de+n%C3%ADveis+de+profici%C3%AAncia+escrita+no+exame+celpe%2Dbras%2E+Master%E2%80%99s+thesis%2C+Universidade+Federal+do+Rio+Grande+do+Sul%2E)
7. INEP. 2020. Documento base do exame Celpe-Bras. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. [find](https://scholar.google.com/scholar?q=INEP%2E+2020%2E+Documento+base+do+exame+Celpe%2DBras%2E+Instituto+Nacional+de+Estudos+e+Pesquisas+Educacionais+An%C3%ADsio+Teixeira%2E)
8. Yingying Liu, Huilei Qi, and Xiaofei Lu. 2025. Enhancing gpt-based automated essay scoring: the impact of fine-tuning and linguistic complexity measures. Computer Assisted Language Learning, pages 1–20. [find](https://scholar.google.com/scholar?q=Yingying+Liu%2C+Huilei+Qi%2C+and+Xiaofei+Lu%2E+2025%2E+Enhancing+gpt%2Dbased+automated+essay+scoring%3A+the+impact+of+fine%2Dtuning+and+linguistic+complexity+measures%2E+Computer+Assisted+Language+Learning%2C+pages+1%E2%80%932)
9. Josephine Lukito, Bin Chen, Gina M. Masullo, and Natalie Jomini Stroud. 2024. Comparing a BERT classifier and a GPT classifier for detecting connective language across multiple social media. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19140–19153,… [find](https://scholar.google.com/scholar?q=Josephine+Lukito%2C+Bin+Chen%2C+Gina+M%2E+Masullo%2C+and+Natalie+Jomini+Stroud%2E+2024%2E+Comparing+a+BERT+classifier+and+a+GPT+classifier+for+detecting+connective+language+across+multiple+social+media%2E+In+Procee)
10. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. [find](https://scholar.google.com/scholar?q=Atsushi+Mizumoto+and+Masaki+Eguchi%2E+2023%2E+Exploring+the+potential+of+using+an+ai+language+model+for+automated+essay+scoring%2E+Research+Methods+in+Applied+Linguistics%2C+2%282%29%3A100050%2E)
11. Ellen Yurika Nagasawa. 2023. O conteúdo de insumo em tarefas que integram leitura e escrita no celpe-bras: uma abordagem informada por corpus. Doctoral dissertation. [find](https://scholar.google.com/scholar?q=Ellen+Yurika+Nagasawa%2E+2023%2E+O+conte%C3%BAdo+de+insumo+em+tarefas+que+integram+leitura+e+escrita+no+celpe%2Dbras%3A+uma+abordagem+informada+por+corpus%2E+Doctoral+dissertation%2E)
12. Austin Pack, Alex Barrett, and Juan Escalante. 2024. Large language models and automated essay scoring of english language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6:100234. [find](https://scholar.google.com/scholar?q=Austin+Pack%2C+Alex+Barrett%2C+and+Juan+Escalante%2E+2024%2E+Large+language+models+and+automated+essay+scoring+of+english+language+learner+writing%3A+Insights+into+validity+and+reliability%2E+Computers+and+Educat)
13. Youri Peskine, Damir Korenčić, Ivan Grubisic, Paolo Papotti, Raphael Troncy, and Paolo Rosso. 2023. Definitions matter: Guiding gpt for multi-label classification. In EMNLP 2023, Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Youri+Peskine%2C+Damir+Korenc%CC%8Cic%CC%81%2C+Ivan+Grubisic%2C+Paolo+Papotti%2C+Raphael+Troncy%2C+and+Paolo+Rosso%2E+2023%2E+Definitions+matter%3A+Guiding+gpt+for+multi%2Dlabel+classification%2E+In+EMNLP+2023%2C+Conference+on+Empir)
14. Marcos Piau, Roberto Lotufo, and Rodrigo Nogueira. 2024. ptt5-v2: A closer look at continued pretraining of t5 models for the portuguese language. [find](https://scholar.google.com/scholar?q=Marcos+Piau%2C+Roberto+Lotufo%2C+and+Rodrigo+Nogueira%2E+2024%2E+ptt5%2Dv2%3A+A+closer+look+at+continued+pretraining+of+t5+models+for+the+portuguese+language%2E)
15. Amanda Pontes Rassi and Priscilla de Abreu Lopes. 2024. Correção automática de redação. In Helena de Medeiros Caseli and Maria das Graças Volpe Nunes, editors, Processamento de Linguagem Natural: conceitos, técnicas e aplicações em português, 3 edition, pages 610–633. BPLN, São Carlos. [find](https://scholar.google.com/scholar?q=Amanda+Pontes+Rassi+and+Priscilla+de+Abreu+Lopes%2E+2024%2E+Corre%C3%A7%C3%A3o+autom%C3%A1tica+de+reda%C3%A7%C3%A3o%2E+In+Helena+de+Medeiros+Caseli+and+Maria+das+Gra%C3%A7as+Volpe+Nunes%2C+editors%2C+Processamento+de+Linguagem+Natural%3A+conc)
16. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. 2024. Avaliação automática do nível de complexidade de textos em português europeu. Linguamática, 16(2):115–139. [find](https://scholar.google.com/scholar?q=Eug%C3%A9nio+Ribeiro%2C+Nuno+Mamede%2C+and+Jorge+Baptista%2E+2024%2E+Avalia%C3%A7%C3%A3o+autom%C3%A1tica+do+n%C3%ADvel+de+complexidade+de+textos+em+portugu%C3%AAs+europeu%2E+Linguam%C3%A1tica%2C+16%282%29%3A115%E2%80%93139%2E)
17. João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023. Advancing neural encoding of portuguese with transformer albertina pt. In EPIA Conference on Artificial Intelligence, pages 441–453. Springer. [find](https://scholar.google.com/scholar?q=Jo%C3%A3o+Rodrigues%2C+Lu%C3%ADs+Gomes%2C+Jo%C3%A3o+Silva%2C+Ant%C3%B3nio+Branco%2C+Rodrigo+Santos%2C+Henrique+Lopes+Cardoso%2C+and+Tom%C3%A1s+Os%C3%B3rio%2E+2023%2E+Advancing+neural+encoding+of+portuguese+with+transformer+albertina+pt%2E+In+EPIA+C)
18. Juliana Schoffen, Margarete Schlatter, Simone Paula Kunrath, Ellen Yurika Nagasawa, Gabrielle Rodrigues Sirianni, Kaiane Mendel, Luana Ramos Truyllio, and Luiza Sarmento Divino. 2018. Estudo descritivo das tarefas da parte escrita do exame celpe-bras: Edições de 1998 a 2017. Technical report, Porto… [find](https://scholar.google.com/scholar?q=Juliana+Schoffen%2C+Margarete+Schlatter%2C+Simone+Paula+Kunrath%2C+Ellen+Yurika+Nagasawa%2C+Gabrielle+Rodrigues+Sirianni%2C+Kaiane+Mendel%2C+Luana+Ramos+Truyllio%2C+and+Luiza+Sarmento+Divino%2E+2018%2E+Estudo+descritiv)
19. Juliana Roquele Schoffen, Elisa Marchioro Stumpf, Luiza Sarmento Divino, Isadora Dahmer Hanauer, Deise Amaral, Amanda Michel Raupp, and Brenda de Souza Xavier. 2025. Corcel: A brazilian portuguese \\<i\>corpus\\</i\> of celpe-bras exam written texts. Revista Brasileira de Linguística Aplicada, 25(1):e5003… [find](https://scholar.google.com/scholar?q=Juliana+Roquele+Schoffen%2C+Elisa+Marchioro+Stumpf%2C+Luiza+Sarmento+Divino%2C+Isadora+Dahmer+Hanauer%2C+Deise+Amaral%2C+Amanda+Michel+Raupp%2C+and+Brenda+de+Souza+Xavier%2E+2025%2E+Corcel%3A+A+brazilian+portuguese+%5C%3Ci)
20. Mark D Shermis and Joshua Wilson. 2024. Introduction to automated essay evaluation. In The Routledge international handbook of automated essay evaluation, pages 3–22. Routledge. [find](https://scholar.google.com/scholar?q=Mark+D+Shermis+and+Joshua+Wilson%2E+2024%2E+Introduction+to+automated+essay+evaluation%2E+In+The+Routledge+international+handbook+of+automated+essay+evaluation%2C+pages+3%E2%80%9322%2E+Routledge%2E)
21. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: pretrained bert models for brazilian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I 9, pages 403–417. Springer. [find](https://scholar.google.com/scholar?q=F%C3%A1bio+Souza%2C+Rodrigo+Nogueira%2C+and+Roberto+Lotufo%2E+2020%2E+Bertimbau%3A+pretrained+bert+models+for+brazilian+portuguese%2E+In+Intelligent+Systems%3A+9th+Brazilian+Conference%2C+BRACIS+2020%2C+Rio+Grande%2C+Brazil%2C+)
22. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. [find](https://scholar.google.com/scholar?q=Yuqing+Tang%2C+Chau+Tran%2C+Xian+Li%2C+Peng%2DJen+Chen%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Jiatao+Gu%2C+and+Angela+Fan%2E+2020%2E+Multilingual+translation+with+extensible+multilingual+pretraining+and+finetuning%2E)
23. Erik Voss. 2025. Comparison of traditional machine learning and neural network approaches for automated scoring of second language english essays. Language Testing, 42(4):369–396. [find](https://scholar.google.com/scholar?q=Erik+Voss%2E+2025%2E+Comparison+of+traditional+machine+learning+and+neural+network+approaches+for+automated+scoring+of+second+language+english+essays%2E+Language+Testing%2C+42%284%29%3A369%E2%80%93396%2E)
24. Jorge A. Wagner Filho, Rodrigo Wilkens, Marco Idiart, and Aline Villavicencio. 2018. The brWaC corpus: A new open resource for Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources… [find](https://scholar.google.com/scholar?q=Jorge+A%2E+Wagner+Filho%2C+Rodrigo+Wilkens%2C+Marco+Idiart%2C+and+Aline+Villavicencio%2E+2018%2E+The+brWaC+corpus%3A+A+new+open+resource+for+Brazilian+Portuguese%2E+In+Proceedings+of+the+Eleventh+International+Confer)
25. Rodrigo Wilkens, Alice Pintard, David Alfter, Vincent Folny, and Thomas François. 2023. Tcfle-8: a corpus of learner written productions for french as a foreign language and its application to automated essay scoring. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro… [find](https://scholar.google.com/scholar?q=Rodrigo+Wilkens%2C+Alice+Pintard%2C+David+Alfter%2C+Vincent+Folny%2C+and+Thomas+Fran%C3%A7ois%2E+2023%2E+Tcfle%2D8%3A+a+corpus+of+learner+written+productions+for+french+as+a+foreign+language+and+its+application+to+automat)
26. Steven Yeung. 2025. A comparative study of rule-based, machine learning and large language model approaches in automated writing evaluation (awe). In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 984–991. [find](https://scholar.google.com/scholar?q=Steven+Yeung%2E+2025%2E+A+comparative+study+of+rule%2Dbased%2C+machine+learning+and+large+language+model+approaches+in+automated+writing+evaluation+%28awe%29%2E+In+Proceedings+of+the+15th+International+Learning+Ana)
27. Ricardo Mazza Zago and Luciane Agnoletti dos Santos Pedotti. 2024. Bertugues: A novel bert transformer model pre-trained for brazilian portuguese. Semina: Ciências Exatas e Tecnológicas, 45:e50630– e50630. [find](https://scholar.google.com/scholar?q=Ricardo+Mazza+Zago+and+Luciane+Agnoletti+dos+Santos+Pedotti%2E+2024%2E+Bertugues%3A+A+novel+bert+transformer+model+pre%2Dtrained+for+brazilian+portuguese%2E+Semina%3A+Ci%C3%AAncias+Exatas+e+Tecnol%C3%B3gicas%2C+45%3Ae50630%E2%80%93+e5)

*Updated 1 November 2026*