---
title: "An Evaluation of Classifiers for Mapping Generative LLM"
description: "They tested four neural designs built on fine-tuned BERT-family encoders. 46 references."
authors: ["Alisea Stroligo", "Anna Shamray", "Julian Schelb", "Andreas Spitz"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.eacl-srw.64/"
doi: "10.18653/v1/2026.eacl-srw.64"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stroligo-2026-evaluation-classifiers-mapping-generative-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stroligo-2026-evaluation-classifiers-mapping-generative-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires

# An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires

**Authors:** Alisea Stroligo, Anna Shamray, [Julian Schelb](https://md.research.elamur.ai/free/en/authors/julian-schelb/) and [Andreas Spitz](https://md.research.elamur.ai/free/en/authors/andreas-spitz/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-srw.64/) · [PDF](https://aclanthology.org/2026.eacl-srw.64.pdf) · [DOI 10.18653/v1/2026.eacl-srw.64](https://doi.org/10.18653/v1/2026.eacl-srw.64) · [OpenAlex](https://openalex.org/W7140146251)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Researchers often have generative LLMs fill in closed-ended questionnaires, for example in persona simulation, machine psychology or benchmark evaluation. The free-form replies then have to be turned back into one of the fixed answer options. This is hard because models add explanations, refuse to answer, or mention several options. The authors compare small, cheap classifiers for this cleaning step.

They tested four neural designs built on fine-tuned BERT-family encoders. Each design scores every pair of answer option and LLM reply. An extra [None] outcome covers invalid replies, detected either by an entropy threshold or by a dedicated class. The baseline is a simple string-matching rule. Evaluation used a new manually annotated set of replies from several state-of-the-art LLMs, drawn from three sources whose answer options differ in complexity.

Unexpectedly, no fine-tuned model beat the rule-based baseline, even on the harder option formats. The best neural model only matched the baseline on one dataset. For standard multiple-choice settings, plain pattern matching therefore remains a competitive option that costs far less compute.

## Key points
- **Datasets:** questions come from the Regulatory Focus Questionnaire (RFQ, Likert-style), ARC and MMLU-Pro. The manually annotated LLM responses, together with the code, are released on GitHub.
- **Rule-based classifier (RbC):** it lower-cases the reply, strips punctuation and splits options into tokens such as "3" and "sometimes". Each option gets a fractional match score capped at 1, and the highest score wins. No match or a tie yields [None].
- **Neural variants:**
  - Three entropy-based 1-vs-all models, including EM-O, which is adapted from Schelb et al. (2025).
  - NEM, which uses an explicit [None] class.
- **Main result:** none of the variants outperformed RbC, despite more than 35,000 training samples. DistilRoBERTa only equalled RbC accuracy on RFQ, and RoBERTa offered the best cost-performance among the encoders.
- **Handling invalid replies:** the entropy-based models had trouble recognising [None] replies on ARC and MMLU-Pro. NEM handled them much better, especially on MMLU-Pro, which has the most invalid replies. Smaller answering LLMs also produced noisier outputs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Konstanz](https://md.research.elamur.ai/free/en/universities/university-of-konstanz/) — University, Konstanz, Germany · 17 works · 232 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Alisea Stroligo, Anna Shamray, Julian Schelb and Andreas Spitz. 2026. An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{stroligo2026evaluation,
  title = {{An Evaluation of Classifiers for Mapping Generative LLM Responses to Answer Options of Multiple-choice Questionnaires}},
  author = {Alisea Stroligo, Anna Shamray, Julian Schelb and Andreas Spitz},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.64},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.64}
}
```

- [← previous An Enduring Enigma: Shedding Light on the Text of the Leiden Riddle, 1845–2025](https://md.research.elamur.ai/free/en/papers/porck-2026-enduring-enigma-shedding-light-text/)
- [next → An Investigation of Linguistic Biases in LLM-Based Recommendations](https://md.research.elamur.ai/free/en/papers/venkateswaran-2026-investigation-linguistic-biases-llm-based/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/)

## References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, ICML, pages 337–371. [find](https://scholar.google.com/scholar?q=Gati+V%2E+Aher%2C+Rosa+I%2E+Arriaga%2C+and+Adam+Tauman+Kalai%2E+2023%2E+Using+large+language+models+to+simulate+multiple+humans+and+replicate+human+subject+studies%2E+In+International+Conference+on+Machine+Learning)
2. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. [Out of one, many: Using language models to simulate human samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). Political Analysis, 31(3):337–351.
3. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. [find](https://scholar.google.com/scholar?q=Marcel+Binz+and+Eric+Schulz%2E+2023%2E+Using+cognitive+psychology+to+understand+GPT%2D3%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+120%286%29%3Ae2218523120%2E)
4. Bojana Bodroža, Bojana M Dinić, and Ljubiša Bojić. 2024. Personality testing of large language models: limited temporal stability, but highlighted prosociality. Royal Society Open Science, 11(10):240180. [find](https://scholar.google.com/scholar?q=Bojana+Bodro%C5%BEa%2C+Bojana+M+Dinic%CC%81%2C+and+Ljubi%C5%A1a+Bojic%CC%81%2E+2024%2E+Personality+testing+of+large+language+models%3A+limited+temporal+stability%2C+but+highlighted+prosociality%2E+Royal+Society+Open+Science%2C+11%2810%29%3A24)
5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? Try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457.
6. Washington Cunha, Leonardo Rocha, and Marcos André Gonçalves. 2025. A thorough benchmark of automatic text classification: From traditional approaches to large language models. CoRR, abs/2504.01930. — [arXiv:2504.01930](https://arxiv.org/abs/2504.01930) [find](https://scholar.google.com/scholar?q=Washington+Cunha%2C+Leonardo+Rocha%2C+and+Marcos+Andr%C3%A9+Gon%C3%A7alves%2E+2025%2E+A+thorough+benchmark+of+automatic+text+classification%3A+From+traditional+approaches+to+large+language+models%2E+CoRR%2C+abs%2F2504%2E01930%2E+%E2%80%94)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805.
8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). CoRR, abs/2407.21783.
9. John Fields, Kevin Chovanec, and Praveen Madiraju. 2024. A survey of text classification with transformers: How wide? How large? How long? How accurate? How expensive? How safe? IEEE Access, 12:6518–6531. [find](https://scholar.google.com/scholar?q=John+Fields%2C+Kevin+Chovanec%2C+and+Praveen+Madiraju%2E+2024%2E+A+survey+of+text+classification+with+transformers%3A+How+wide%3F+How+large%3F+How+long%3F+How+accurate%3F+How+expensive%3F+How+safe%3F+IEEE+Access%2C+12%3A6518%E2%80%936)
10. Andrea Gasparetto, Matteo Marcuzzo, Alessandro Zangari, and Andrea Albarelli. 2022. A survey on text classification algorithms: From text to predictions. Inf., 13(2):83. [find](https://scholar.google.com/scholar?q=Andrea+Gasparetto%2C+Matteo+Marcuzzo%2C+Alessandro+Zangari%2C+and+Andrea+Albarelli%2E+2022%2E+A+survey+on+text+classification+algorithms%3A+From+text+to+predictions%2E+Inf%2E%2C+13%282%29%3A83%2E)
11. Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. 2013. BigBench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD ’13, page 1197–1208… [find](https://scholar.google.com/scholar?q=Ahmad+Ghazal%2C+Tilmann+Rabl%2C+Minqing+Hu%2C+Francois+Raab%2C+Meikel+Poess%2C+Alain+Crolotte%2C+and+Hans%2DArno+Jacobsen%2E+2013%2E+BigBench%3A+Towards+an+industry+standard+benchmark+for+big+data+analytics%2E+In+Proceedin)
12. Hyukjun Gweon and Matthias Schonlau. 2023. Automated classification for open-ended questions with BERT. Journal of Survey Statistics and Methodology, 12(2):493–504. [find](https://scholar.google.com/scholar?q=Hyukjun+Gweon+and+Matthias+Schonlau%2E+2023%2E+Automated+classification+for+open%2Dended+questions+with+BERT%2E+Journal+of+Survey+Statistics+and+Methodology%2C+12%282%29%3A493%E2%80%93504%2E)
13. Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. Machine psychology. Preprint, arXiv:2303.13988. — [arXiv:2303.13988](https://arxiv.org/abs/2303.13988) [find](https://scholar.google.com/scholar?q=Thilo+Hagendorff%2C+Ishita+Dasgupta%2C+Marcel+Binz%2C+Stephanie+C%2E+Y%2E+Chan%2C+Andrew+Lampinen%2C+Jane+X%2E+Wang%2C+Zeynep+Akata%2C+and+Eric+Schulz%2E+2024%2E+Machine+psychology%2E+Preprint%2C+arXiv%3A2303%2E13988%2E+%E2%80%94+%5BarXiv%3A2303%2E)
14. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
15. Rohit Kumar Kaliyar, Anurag Goswami, and Pratik Narang. 2021. FakeBERT: Fake news detection in social media with a BERT-based deep learning approach. Multim. Tools Appl., 80(8):11765–11788. [find](https://scholar.google.com/scholar?q=Rohit+Kumar+Kaliyar%2C+Anurag+Goswami%2C+and+Pratik+Narang%2E+2021%2E+FakeBERT%3A+Fake+news+detection+in+social+media+with+a+BERT%2Dbased+deep+learning+approach%2E+Multim%2E+Tools+Appl%2E%2C+80%288%29%3A11765%E2%80%9311788%2E)
16. Arina Kostina, Marios D. Dikaiakos, Dimosthenis Stefanidis, and George Pallis. 2025. Large language models for text classification: Case study and comprehensive review. CoRR, abs/2501.08457. — [arXiv:2501.08457](https://arxiv.org/abs/2501.08457) [find](https://scholar.google.com/scholar?q=Arina+Kostina%2C+Marios+D%2E+Dikaiakos%2C+Dimosthenis+Stefanidis%2C+and+George+Pallis%2E+2025%2E+Large+language+models+for+text+classification%3A+Case+study+and+comprehensive+review%2E+CoRR%2C+abs%2F2501%2E08457%2E+%E2%80%94+%5BarXiv%3A)
17. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. [ALBERT: A lite BERT for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi…
18. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025. From generation to judgment: Opportunities and challenges of LLM-as-a-judge. In Proceedings of the 2025 Conference o… [find](https://scholar.google.com/scholar?q=Dawei+Li%2C+Bohan+Jiang%2C+Liangjie+Huang%2C+Alimohammad+Beigi%2C+Chengshuai+Zhao%2C+Zhen+Tan%2C+Amrita+Bhattacharjee%2C+Yuxuan+Jiang%2C+Canyu+Chen%2C+Tianhao+Wu%2C+Kai+Shu%2C+Lu+Cheng%2C+and+Huan+Liu%2E+2025%2E+From+generation+)
19. Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, and Lifang He. 2022. A survey on text classification: From traditional to deep learning. ACM Trans. Intell. Syst. Technol., 13(2). [find](https://scholar.google.com/scholar?q=Qian+Li%2C+Hao+Peng%2C+Jianxin+Li%2C+Congying+Xia%2C+Renyu+Yang%2C+Lichao+Sun%2C+Philip+S%2E+Yu%2C+and+Lifang+He%2E+2022%2E+A+survey+on+text+classification%3A+From+traditional+to+deep+learning%2E+ACM+Trans%2E+Intell%2E+Syst%2E+Tec)
20. Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. 2024. Evaluating psychological safety of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1826–1843, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xingxuan+Li%2C+Yutong+Li%2C+Lin+Qiu%2C+Shafiq+Joty%2C+and+Lidong+Bing%2E+2024%2E+Evaluating+psychological+safety+of+large+language+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Lan)
21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). CoRR, abs/2009.03300.
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). CoRR, abs/1907.11692.
23. E. Tory Higgins, Ronald S. Friedman, Robert E. Harlow, Lorraine Chen Idson, Ozlem N. Ayduk, and Amy Taylor. 2001. Achievement orientations from subjective histories of success: Promotion pride versus prevention pride. European Journal of Social Psychology, 31(1):3–23. [find](https://scholar.google.com/scholar?q=E%2E+Tory+Higgins%2C+Ronald+S%2E+Friedman%2C+Robert+E%2E+Harlow%2C+Lorraine+Chen+Idson%2C+Ozlem+N%2E+Ayduk%2C+and+Amy+Taylor%2E+2001%2E+Achievement+orientations+from+subjective+histories+of+success%3A+Promotion+pride+versus+)
24. Xinyi Lu and Xu Wang. 2024. Generative students: Using LLM-simulated student profiles to support question item evaluation. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, page 16–27, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Xinyi+Lu+and+Xu+Wang%2E+2024%2E+Generative+students%3A+Using+LLM%2Dsimulated+student+profiles+to+support+question+item+evaluation%2E+In+Proceedings+of+the+Eleventh+ACM+Conference+on+Learning+%40+Scale%2C+L%40S+%E2%80%9924%2C+p)
25. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research. [find](https://scholar.google.com/scholar?q=John+J+Horton%2E+2023%2E+Large+language+models+as+simulated+economic+agents%3A+What+can+we+learn+from+homo+silicus%3F+Working+Paper+31122%2C+National+Bureau+of+Economic+Research%2E)
26. Yang Lu, Jordan Yu, and Shou-Hsuan Stephen Huang. 2023. Illuminating the black box: A psychometric investigation into the multifaceted nature of large language models. Preprint, arXiv:2312.14202. — [arXiv:2312.14202](https://arxiv.org/abs/2312.14202) [find](https://scholar.google.com/scholar?q=Yang+Lu%2C+Jordan+Yu%2C+and+Shou%2DHsuan+Stephen+Huang%2E+2023%2E+Illuminating+the+black+box%3A+A+psychometric+investigation+into+the+multifaceted+nature+of+large+language+models%2E+Preprint%2C+arXiv%3A2312%2E14202%2E+%E2%80%94+%5Ba)
27. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2022. Deep learning-based text classification: A comprehensive review. ACM Comput. Surv., 54(3):62:1–62:40. [find](https://scholar.google.com/scholar?q=Shervin+Minaee%2C+Nal+Kalchbrenner%2C+Erik+Cambria%2C+Narjes+Nikzad%2C+Meysam+Chenaghlu%2C+and+Jianfeng+Gao%2E+2022%2E+Deep+learning%2Dbased+text+classification%3A+A+comprehensive+review%2E+ACM+Comput%2E+Surv%2E%2C+54%283%29%3A62%3A1%E2%80%93)
28. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? An exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS). [find](https://scholar.google.com/scholar?q=Maril%C3%B9+Miotto%2C+Nicola+Rossberg%2C+and+Bennett+Kleinberg%2E+2022%2E+Who+is+GPT%2D3%3F+An+exploration+of+personality%2C+values+and+demographics%2E+In+Proceedings+of+the+Fifth+Workshop+on+Natural+Language+Processing+a)
29. Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19(5):808–826. [find](https://scholar.google.com/scholar?q=Max+Pellert%2C+Clemens+M+Lechner%2C+Claudia+Wagner%2C+Beatrice+Rammstedt%2C+and+Markus+Strohmaier%2E+2024%2E+AI+psychometrics%3A+Assessing+the+psychological+profiles+of+large+language+models+through+psychometric+in)
30. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+CoRR%2C+abs%2F1910%2E01108%2E)
32. Julian Schelb, Orr Borin, David Garcia, and Andreas Spitz. 2025. R.u.psycho? Robust unified psychometric testing of language models. Preprint, arXiv:2503.10229. — [arXiv:2503.10229](https://arxiv.org/abs/2503.10229) [find](https://scholar.google.com/scholar?q=Julian+Schelb%2C+Orr+Borin%2C+David+Garcia%2C+and+Andreas+Spitz%2E+2025%2E+R%2Eu%2Epsycho%3F+Robust+unified+psychometric+testing+of+language+models%2E+Preprint%2C+arXiv%3A2503%2E10229%2E+%E2%80%94+%5BarXiv%3A2503%2E10229%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
33. Matthias Schonlau, Julia Weiß, and Jan Marquardt. 2023. Multi-label classification of open-ended questions with BERT. CoRR, abs/2304.02945. [find](https://scholar.google.com/scholar?q=Matthias+Schonlau%2C+Julia+Wei%C3%9F%2C+and+Jan+Marquardt%2E+2023%2E+Multi%2Dlabel+classification+of+open%2Dended+questions+with+BERT%2E+CoRR%2C+abs%2F2304%2E02945%2E)
34. Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2025. A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence, 7(12):1954– 1968. [find](https://scholar.google.com/scholar?q=Gregory+Serapio%2DGarc%C3%ADa%2C+Mustafa+Safdari%2C+Cl%C3%A9ment+Crepy%2C+Luning+Sun%2C+Stephen+Fitz%2C+Peter+Romero%2C+Marwa+Abdulhai%2C+Aleksandra+Faust%2C+and+Maja+Mataric%CC%81%2E+2025%2E+A+psychometric+framework+for+evaluating+and+s)
35. S. S. Stevens. 1946. On the theory of scales of measurement. Science, 103(2684):677–680. [find](https://scholar.google.com/scholar?q=S%2E+S%2E+Stevens%2E+1946%2E+On+the+theory+of+scales+of+measurement%2E+Science%2C+103%282684%29%3A677%E2%80%93680%2E)
36. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. [Text classification via large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-text-classification-large-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005, Singapore. Association for Computational Linguistics.
37. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+Louis+Rouillard%2C+Thomas+Mesnard%2C+Geoffrey)
38. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
39. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, and et al. 2023. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.16944. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+and+et+al%2E+2023%2E+Zephyr%3A+Direct+distillation+of+LM+alignment%2E+CoRR%2C+abs%2F2310%2E16944%2E)
40. Sowmya Vajjala and Shwetali Shimangaud. 2025. Text classification in the LLM era - Where do we stand? CoRR, abs/2502.11830. — [arXiv:2502.11830](https://arxiv.org/abs/2502.11830) [find](https://scholar.google.com/scholar?q=Sowmya+Vajjala+and+Shwetali+Shimangaud%2E+2025%2E+Text+classification+in+the+LLM+era+%2D+Where+do+we+stand%3F+CoRR%2C+abs%2F2502%2E11830%2E+%E2%80%94+%5BarXiv%3A2502%2E11830%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E11830%29)
41. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
42. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024a. ["My Answer is C": First-token probabilities do not match text answers in instruction-tuned language models](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/). In Findings of the Association for Computational Linguistics, ACL.
43. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. [MMLU-Pro: A more robust and challenging multi-task language understanding benchmar…](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
44. Hanzi Xu, Renze Lou, Jiangshu Du, Vahid Mahzoon, Elmira Talebianaraki, Zhuoan Zhou, Elizabeth Garrison, Slobodan Vucetic, and Wenpeng Yin. 2024. LLMs’ classification performance is overclaimed. CoRR, abs/2406.16203. — [arXiv:2406.16203](https://arxiv.org/abs/2406.16203) [find](https://scholar.google.com/scholar?q=Hanzi+Xu%2C+Renze+Lou%2C+Jiangshu+Du%2C+Vahid+Mahzoon%2C+Elmira+Talebianaraki%2C+Zhuoan+Zhou%2C+Elizabeth+Garrison%2C+Slobodan+Vucetic%2C+and+Wenpeng+Yin%2E+2024%2E+LLMs%E2%80%99+classification+performance+is+overclaimed%2E+CoRR%2C+)
45. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and et al. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+and+et+al%2E+2024%2E+Qwen2%2E5+technical+report%2E+CoRR%2C+abs%2F2412%2E15115%2E)
46. Yazhou Zhang, Mengyao Wang, Qiuchi Li, Prayag Tiwari, and Jing Qin. 2025. Pushing the limit of LLM capacity for text classification. In Companion Proceedings of the ACM on Web Conference 2025, WWW ’25, page 1524–1528, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yazhou+Zhang%2C+Mengyao+Wang%2C+Qiuchi+Li%2C+Prayag+Tiwari%2C+and+Jing+Qin%2E+2025%2E+Pushing+the+limit+of+LLM+capacity+for+text+classification%2E+In+Companion+Proceedings+of+the+ACM+on+Web+Conference+2025%2C+WWW+%E2%80%9925)

*Updated 1 November 2026*