--- title: "EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering — Paiola 2026" description: "A three-step pipeline builds the dataset. 17 references." authors: ["Pedro Henrique Paiola", "Luís Gabriel Damiati Mendes", "Bruno de Oliveira Monchelato", "André da Fonseca Schuck", "Gabriel Lino Garcia", "Douglas Rodrigues", "Helena de Medeiros Caseli", "João Paulo Papa"] year: 2026 venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference" source: "https://aclanthology.org/2026.lrec-1.360/" doi: "10.63317/4nocig8f36r9" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/paiola-2026-edubench-portuguese-benchmark-open-ended/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/paiola-2026-edubench-portuguese-benchmark-open-ended/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering # EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering **Authors:** Pedro Henrique Paiola, Luís Gabriel Damiati Mendes, Bruno de Oliveira Monchelato, André da Fonseca Schuck, Gabriel Lino Garcia, Douglas Rodrigues, Helena de Medeiros Caseli and João Paulo Papa · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.360/) · [PDF](https://aclanthology.org/2026.lrec-1.360.pdf) · [DOI 10.63317/4nocig8f36r9](https://doi.org/10.63317/4nocig8f36r9) · [OpenAlex](https://openalex.org/W7160195651) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Most LLM benchmarks, particularly for Portuguese, test option selection or short span extraction. They reveal little about whether a model can write an extended, well-argued answer. EduBench fills this gap with 3,149 essay-style questions from the second-phase entrance exams of UNESP, USP and UNICAMP, covering 2015–2025 and spanning Humanities, Exact and Natural Sciences, and Languages. Reference answers come from official commented solutions or, when those are unavailable, from resolutions published by Colégio Objetivo. A three-step pipeline builds the dataset. It extracts text and figures from the exam PDFs, replaces images with GPT-4o-mini captions so text-only models can be tested, and uses GPT-4o to segment exams into items, clean prompts, attach answers and assign subject labels. Quality checks follow. Nine models, from 4B-parameter SLMs to reasoning-oriented LLMs, were scored with BLEU, BERTScore and G-Eval. The metrics frequently contradict one another, and models reach roughly 54–71% agreement with expert answers. This shows how hard it is to judge discursive output automatically. The release gives Portuguese NLP a public testbed for long-form generation and its evaluation. ## Key points - **Dataset scope:** 3,149 open-ended questions from Brazilian university entrance exams (UNESP, USP, UNICAMP, 2015–2025), each with institution, year and subject metadata, plus short-form answers where applicable. - **Construction:** GPT-4o-mini generated the figure descriptions and GPT-4o handled structuring. Reference answers were taken from official or Colégio Objetivo commented solutions. - **Baselines:** Nine models ranging from 4B SLMs to state-of-the-art reasoning LLMs were evaluated with BLEU, BERTScore and G-Eval. Alignment with expert answers ranged from 54% to 71%, and the metrics showed notable disagreement. - **Availability:** The data is released on Hugging Face (`recogna-nlp/EduBench`) and the code on GitHub (`pedropaiola/EduBench`). - **Limitations:** The benchmark covers Brazilian Portuguese only and may contain residual extraction errors. Evaluation is fully automatic, with no large-scale human study, which the authors list as future work alongside expansion to other Portuguese varieties. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Cite Cite as: Pedro Henrique Paiola, Luís Gabriel Damiati Mendes, Bruno de Oliveira Monchelato, André da Fonseca Schuck, Gabriel Lino Garcia, Douglas Rodrigues, Helena de Medeiros Caseli and João Paulo Papa. 2026. EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering. Proceedings of the Fifteenth Language Resources and Evaluation Conference. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{paiola2026edubench, title = {{EduBench: A Portuguese Benchmark for Open-Ended Discursive Question Answering}}, author = {Pedro Henrique Paiola, Luís Gabriel Damiati Mendes, Bruno {de Oliveira Monchelato}, André {da Fonseca Schuck}, Gabriel Lino Garcia, Douglas Rodrigues, Helena {de Medeiros Caseli} and João Paulo Papa}, booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference}, year = {2026}, doi = {10.63317/4nocig8f36r9}, url = {https://doi.org/10.63317/4nocig8f36r9} } ``` - [← previous EDRAC: Benchmarking Arabic Dialect Reading Comprehension](https://md.research.elamur.ai/free/en/papers/mokh-2026-edrac-benchmarking-arabic-dialect-reading/) - [next → Efficient Topic Extraction via Graph-Based Labeling: A Lightweight Alternative to Deep Models](https://md.research.elamur.ai/free/en/papers/mekaoui-2026-efficient-topic-extraction-graph-based/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) ## References 17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Thales Sales Almeida, Thiago Laitz, Giovana K. Bonás, and Rodrigo Nogueira. 2023. Bluex: A benchmark based on brazilian leading universities entrance exams. [find](https://scholar.google.com/scholar?q=Thales+Sales+Almeida%2C+Thiago+Laitz%2C+Giovana+K%2E+Bon%C3%A1s%2C+and+Rodrigo+Nogueira%2E+2023%2E+Bluex%3A+A+benchmark+based+on+brazilian+leading+universities+entrance+exams%2E) 2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human generated machine reading comprehension dataset. [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) 3. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. [find](https://scholar.google.com/scholar?q=Guiming+Hardy+Chen%2C+Shunian+Chen%2C+Ziche+Liu%2C+Feng+Jiang%2C+and+Benyou+Wang%2E+2024%2E+Humans+or+LLMs+as+the+judge%3F+a+study+on+judgement+bias%2E) 4. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. [our card](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/) 5. Rudolf Flesch. 1948. A New Readability Yardstick, volume 32. American Psychological Association. [find](https://scholar.google.com/scholar?q=Rudolf+Flesch%2E+1948%2E+A+New+Readability+Yardstick%2C+volume+32%2E+American+Psychological+Association%2E) 6. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/) 7. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+NLG+evaluation+using+GPT%2D4+with+better+human+alignment%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+M) 8. Marcelo Sartori Locatelli et al. 2025. Examining the Behavior of LLM Architectures within the Framework of Standardized National Exams in Brazil, page 879–890. AAAI Press. [find](https://scholar.google.com/scholar?q=Marcelo+Sartori+Locatelli+et+al%2E+2025%2E+Examining+the+Behavior+of+LLM+Architectures+within+the+Framework+of+Standardized+National+Exams+in+Brazil%2C+page+879%E2%80%93890%2E+AAAI+Press%2E) 9. Teresa Bertholdo Freire Martins, Claudete Moreno Ghiraldelo, Maria das Graças Volpe Nunes, and Osvaldo Novais de Oliveira Junior. 1996. Readability Formulas Applied to Textbooks in Brazilian Portuguese. Ph.D. thesis, Instituto de Ciências Matemáticas de São Carlos, Universidade de São Paulo, São Car… [find](https://scholar.google.com/scholar?q=Teresa+Bertholdo+Freire+Martins%2C+Claudete+Moreno+Ghiraldelo%2C+Maria+das+Gra%C3%A7as+Volpe+Nunes%2C+and+Osvaldo+Novais+de+Oliveira+Junior%2E+1996%2E+Readability+Formulas+Applied+to+Textbooks+in+Brazilian+Portugues) 10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 12. Hélio Fonseca Sayama, Anderson Viçoso Araujo, and Eraldo Rezende Fernandes. 2019. Faquad: Reading comprehension dataset in the domain of brazilian higher education. In 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pages 443–448. [find](https://scholar.google.com/scholar?q=H%C3%A9lio+Fonseca+Sayama%2C+Anderson+Vi%C3%A7oso+Araujo%2C+and+Eraldo+Rezende+Fernandes%2E+2019%2E+Faquad%3A+Reading+comprehension+dataset+in+the+domain+of+brazilian+higher+education%2E+In+2019+8th+Brazilian+Conference+on) 13. Aarohi Srivastava et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Featured Certification. [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) 14. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark… [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/) 15. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTscore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTscore%3A+Evaluating+text+generation+with+BERT%2E+In+Proceedings+of+the+8th+International+Conference+on+Learning+Repr) 16. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+llm%2Das) 17. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/)