---
title: "Benchmarking LLMs for Aspect-Based Sentiment Classification in"
description: "The paper asks whether instruction-tuned LLMs, used without supervised fine-tuning, can judge sentiment toward specific… 23 references."
authors: ["Tina Munda", "Filip Dobranić", "Uroš Šmajdek", "Oliver Pejić", "Ciril Bohak", "Vojko Gorjanc", "Darja Fišer"]
year: 2026
venue: "Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026"
source: "https://aclanthology.org/2026.llms4ssh-1.11/"
doi: "10.63317/22hvcbc23rts"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/munda-2026-benchmarking-llms-aspect-based-sentiment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/munda-2026-benchmarking-llms-aspect-based-sentiment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals

# Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals

**Authors:** [Tina Munda](https://md.research.elamur.ai/free/en/authors/tina-munda/), Filip Dobranić, Uroš Šmajdek, Oliver Pejić, Ciril Bohak, Vojko Gorjanc and Darja Fišer · **Year:** 2026 · **Venue:** [Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/llms4ssh/) ([Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities @ LREC 2026](https://md.research.elamur.ai/free/en/venues/llms4ssh/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.llms4ssh-1.11/) · [PDF](https://aclanthology.org/2026.llms4ssh-1.11.pdf) · [DOI 10.63317/22hvcbc23rts](https://doi.org/10.63317/22hvcbc23rts) · [OpenAlex](https://openalex.org/W7166523245)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether instruction-tuned LLMs, used without supervised fine-tuning, can judge sentiment toward specific targets in noisy OCR text from historical Slovene newspapers. Labelled data for this setting is scarce, and the language is both morphologically rich and historically variable. The authors draw texts from three sPeriodika newspapers. Using a hand-built lexicon, they extract nominal and adjectival mentions of collective identities (ethno-national, regional and geographic groups). They then manually annotate 371 mentions for sentiment toward each marked target.

Several models are compared in a few-shot setup using standard classification metrics:
- the Slovene-adapted GaMS3-12B-Instruct;
- general-purpose models of similar size;
- two smaller variants.

The analysis also breaks down performance by grammatical form and by referential type. This reveals systematic asymmetries that skew aggregated results. A dataset-scale aggregation with the best model is offered only as a demonstration of scalability. The authors conclude that aggregated polarity should be read as a directional tendency rather than a precise measurement. They also argue that interpretative validity matters more to digital humanities work than raw F1 scores.

## Key points
- **Data:** Slovenski narod (~183M tokens), Slovenec (~137M) and Slovenka (~1.6M), over 320M tokens in total. Only issues up to 31 December 1914 are included, for copyright reasons.
- **Benchmark:** 371 manually annotated collective-identity mentions, framed as aspect-level (targeted) sentiment classification.
- **Models:**
  - GaMS3-12B-Instruct (Gemma 3-based, continually pretrained on Slovene);
  - Gemma-3-12B-IT, LLaMA 3.1 and DeepSeek-R1-Distill-Qwen-14B;
  - the smaller Gemma-3-4B-IT and DeepSeek-R1-Distill-Qwen-7B, included to test size effects.
- **Lexicon construction:**
  - Noun lemmas were compiled manually from Slovenka.
  - Adjective candidates were found via suffixes such as -ski/-ški, including the historical spellings -zki/-žki. In the two larger papers, they were filtered at a minimum of 90 occurrences and then reviewed.
  - Adjectives were mapped to nominal identity labels (e.g., nemški → Nemci).
- **Diagnostics:** Accuracy differs by realisation (noun vs. adjective) and by referential type (collective actor vs. non-agentive entity). These differences must be accounted for when aggregating sentiment across a corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Ljubljana](https://md.research.elamur.ai/free/en/universities/university-of-ljubljana/) — University, Ljubljana, Slovenia · 13 works · 38,693 citations
- [Institute of Contemporary History](https://md.research.elamur.ai/free/en/labs/institute-of-contemporary-history/) — Research institute, Ljubljana, Slovenia · 4 works · 32 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 254 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 541 works

🧭 Topics (1): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works

## Cite

Cite as: Tina Munda, Filip Dobranić, Uroš Šmajdek, Oliver Pejić, Ciril Bohak, Vojko Gorjanc and Darja Fišer. 2026. Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals. Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{munda2026benchmarking,
  title = {{Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals}},
  author = {Tina Munda, Filip Dobranić, Uroš Šmajdek, Oliver Pejić, Ciril Bohak, Vojko Gorjanc and Darja Fišer},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  year = {2026},
  doi = {10.63317/22hvcbc23rts},
  url = {https://doi.org/10.63317/22hvcbc23rts}
}
```

- [← previous Benchmarking LLM's Capability in Reasoning over Conflicting Web References](https://md.research.elamur.ai/free/en/papers/yuan-2026-benchmarking-llm-s-capability-reasoning/)
- [next → Benchmarking NLP-supported Language Sample Analysis for Swiss Children's Speech](https://md.research.elamur.ai/free/en/papers/ryser-2026-benchmarking-nlp-supported-language-sample/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. ACM. [find](https://scholar.google.com/scholar?q=the+tenth+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pages+168%E2%80%93177%2E+ACM%2E)
2. Swoichha Adhikari, Manan Gangwani, and Adithi Varadarajan. 2024. Aspect-based sentiment analysis for slovene texts: Models, lexicons, and embeddings. In 2024 IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI), volume 2, pages 1–6… [find](https://scholar.google.com/scholar?q=Swoichha+Adhikari%2C+Manan+Gangwani%2C+and+Adithi+Varadarajan%2E+2024%2E+Aspect%2Dbased+sentiment+analysis+for+slovene+texts%3A+Models%2C+lexicons%2C+and+embeddings%2E+In+2024+IEEE+International+Conference+on+Interdisc)
3. Erik Cambria, Björn Schuller, Yunqing Xia, and Catherine Havasi. 2017. New avenues in opinion mining and sentiment analysis. IEEE Intelligent Systems, 28(2):15–21. [find](https://scholar.google.com/scholar?q=Erik+Cambria%2C+Bj%C3%B6rn+Schuller%2C+Yunqing+Xia%2C+and+Catherine+Havasi%2E+2017%2E+New+avenues+in+opinion+mining+and+sentiment+analysis%2E+IEEE+Intelligent+Systems%2C+28%282%29%3A15%E2%80%9321%2E)
4. William B Cavnar, John M Trenkle, et al. 1994. Ngram-based text categorization. In Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval, volume 161175, page 14. Las Vegas, NV. [find](https://scholar.google.com/scholar?q=William+B+Cavnar%2C+John+M+Trenkle%2C+et+al%2E+1994%2E+Ngram%2Dbased+text+categorization%2E+In+Proceedings+of+SDAIR%2D94%2C+3rd+annual+symposium+on+document+analysis+and+information+retrieval%2C+volume+161175%2C+page+14%2E)
5. Corinna Cortes and Vladimir Vapnik. 1995. [Support-vector networks](https://md.research.elamur.ai/free/en/papers/cortes-1995-support-vector-networks/). Machine learning, 20(3):273–297.
6. DeepSeek-AI. 2025. [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/).
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
8. Filip Dobranić, Bojan Evkoski, and Nikola Ljubešić. 2024. A lightweight approach to a giga-corpus of historical periodicals: The story of a slovenian historical newspaper collection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalua… [find](https://scholar.google.com/scholar?q=Filip+Dobrani%C4%87%2C+Bojan+Evkoski%2C+and+Nikola+Ljube%C5%A1i%C4%87%2E+2024%2E+A+lightweight+approach+to+a+giga%2Dcorpus+of+historical+periodicals%3A+The+story+of+a+slovenian+historical+newspaper+collection%2E+In+Proceedings+of)
9. Darja Fišer, Jasmina Smailovic, Tomaž Erjavec, Igor Mozetic, and Miha Grcar. 2016. Sentiment annotation of slovene user-generated content. In Proceedings of the 2016 conference language technologies and digital humanities (JTDH 2016), pages 65–70. [find](https://scholar.google.com/scholar?q=Darja+Fi%C5%A1er%2C+Jasmina+Smailovic%2C+Toma%C5%BE+Erjavec%2C+Igor+Mozetic%2C+and+Miha+Grcar%2E+2016%2E+Sentiment+annotation+of+slovene+user%2Dgenerated+content%2E+In+Proceedings+of+the+2016+conference+language+technologies+a)
10. Aaron Grattafiori and Abhimanyu Dubey et al. 2024. [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/).
11. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of C.J. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+C%2EJ%2E)
12. Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth International Conference on Weblogs and Social Media (ICWSM-14), pages 216–225. [find](https://scholar.google.com/scholar?q=Hutto+and+Eric+Gilbert%2E+2014%2E+Vader%3A+A+parsimonious+rule%2Dbased+model+for+sentiment+analysis+of+social+media+text%2E+In+Eighth+International+Conference+on+Weblogs+and+Social+Media+%28ICWSM%2D14%29%2C+pages+216%E2%80%932)
13. Marjeta Keršič-Svetel. 1996. Češko-slovenski stiki med svetovnima vojnama. Zbirka Zgodovinskega časopisa. Zveza zgodovinskih društev Slovenije, Ljubljana. [find](https://scholar.google.com/scholar?q=Marjeta+Ker%C5%A1i%C4%8D%2DSvetel%2E+1996%2E+%C4%8Ce%C5%A1ko%2Dslovenski+stiki+med+svetovnima+vojnama%2E+Zbirka+Zgodovinskega+%C4%8Dasopisa%2E+Zveza+zgodovinskih+dru%C5%A1tev+Slovenije%2C+Ljubljana%2E)
14. Oto Luthar, Igor Grdina, Marjeta Šašel Kos, Petra Svoljšak, Peter Kos, Dušan Kos, Peter Štih, Alja Brglez, and Martin Pogačar. 2008. The land between: a history of slovenia. [find](https://scholar.google.com/scholar?q=Oto+Luthar%2C+Igor+Grdina%2C+Marjeta+%C5%A0a%C5%A1el+Kos%2C+Petra+Svolj%C5%A1ak%2C+Peter+Kos%2C+Du%C5%A1an+Kos%2C+Peter+%C5%A0tih%2C+Alja+Brglez%2C+and+Martin+Poga%C4%8Dar%2E+2008%2E+The+land+between%3A+a+history+of+slovenia%2E)
15. Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 19(2):313–330.
16. Iftikhar Muhammad, Marco Rospocher, Timotej Knez, and Slavko Žitnik. 2025. Benchmarking large language models for target-based financial sentiment analysis. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 785–795, Cagliari, Italy. CEUR Workshop Pr… [find](https://scholar.google.com/scholar?q=Iftikhar+Muhammad%2C+Marco+Rospocher%2C+Timotej+Knez%2C+and+Slavko+%C5%BDitnik%2E+2025%2E+Benchmarking+large+language+models+for+target%2Dbased+financial+sentiment+analysis%2E+In+Proceedings+of+the+Eleventh+Italian+Conf)
17. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. [Thumbs up? sentiment classification using machine learning techniques](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/). In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics.
18. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. [Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence](https://md.research.elamur.ai/free/en/papers/sun-2019-utilizing-bert-aspect-based-sentiment/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 380–385.
19. Domen Vreš, Martin Božič, Aljaž Potočnik, Tomaž Martinčič, and Marko Robnik-Šikonja. 2024. Generative model for less-resourced language with 1 billion parameters. arXiv preprint arXiv:2410.06898. — [arXiv:2410.06898](https://arxiv.org/abs/2410.06898) [find](https://scholar.google.com/scholar?q=Domen+Vre%C5%A1%2C+Martin+Bo%C5%BEi%C4%8D%2C+Alja%C5%BE+Poto%C4%8Dnik%2C+Toma%C5%BE+Martin%C4%8Di%C4%8D%2C+and+Marko+Robnik%2D%C5%A0ikonja%2E+2024%2E+Generative+model+for+less%2Dresourced+language+with+1+billion+parameters%2E+arXiv+preprint+arXiv%3A2410%2E06898%2E+%E2%80%94+%5Ba)
20. Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2024. [Sentiment analysis in the era of large language models: A reality check](https://md.research.elamur.ai/free/en/papers/zhang-2023-sentiment-analysis-era-large-language/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881– 3906. Association for Computational Linguistics.
21. Filip Čuček. 2016. Svoji k svojim: na poti k dokončni nacionalni razmejitvi na spodnjem Štajerskem v 19. stoletju. [find](https://scholar.google.com/scholar?q=Filip+%C4%8Cu%C4%8Dek%2E+2016%2E+Svoji+k+svojim%3A+na+poti+k+dokon%C4%8Dni+nacionalni+razmejitvi+na+spodnjem+%C5%A0tajerskem+v+19%2E+stoletju%2E)
22. Slavko Žitnik, Neli Blagus, and Marko Bajec. 2022. Target-level sentiment analysis for news articles. Knowledge-Based Systems, 249:108939. [find](https://scholar.google.com/scholar?q=Slavko+%C5%BDitnik%2C+Neli+Blagus%2C+and+Marko+Bajec%2E+2022%2E+Target%2Dlevel+sentiment+analysis+for+news+articles%2E+Knowledge%2DBased+Systems%2C+249%3A108939%2E)
23. Language Resource References Filip Dobranić, Bojan Evkoski, and Nikola Ljubešić. 2023. Corpus of slovenian periodicals (1771-1914) sPeriodika 1.0. Slovenian language resource repository CLARIN.SI. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Filip+Dobrani%C4%87%2C+Bojan+Evkoski%2C+and+Nikola+Ljube%C5%A1i%C4%87%2E+2023%2E+Corpus+of+slovenian+periodicals+%281771%2D1914%29+sPeriodika+1%2E0%2E+Slovenian+language+resource+repository+CLARIN%2ESI%2E)

*Updated 2 November 2026*