---
title: "Evaluating Multilingual Sentiment Classifiers Using an"
description: "The authors build a multilingual, multi-domain benchmark from the lead sections of important articles in English, German… 15 references."
authors: ["Milena Stróżyna", "Włodzimierz Lewoniewski", "Izabela Czumałowska"]
year: 2026
venue: "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)"
source: "https://aclanthology.org/2026.gem-main.63/"
doi: "10.18653/v1/2026.gem-main.63"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/strozyna-2026-evaluating-multilingual-sentiment-classifiers-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/strozyna-2026-evaluating-multilingual-sentiment-classifiers-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark

# Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark

**Authors:** Milena Stróżyna, Włodzimierz Lewoniewski and Izabela Czumałowska · **Year:** 2026 · **Venue:** [Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)](https://md.research.elamur.ai/free/en/venues/gem/) ([GEM 2026](https://md.research.elamur.ai/free/en/venues/gem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gem-main.63/) · [PDF](https://aclanthology.org/2026.gem-main.63.pdf) · [DOI 10.18653/v1/2026.gem-main.63](https://doi.org/10.18653/v1/2026.gem-main.63) · [OpenAlex](https://openalex.org/W7166817397)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Wikipedia articles must follow the Neutral Point of View (NPOV) policy, so any sentiment they carry is rare and implicit. This makes them a hard test for sentiment tools, especially when comparing language editions. The authors build a multilingual, multi-domain benchmark from the lead sections of important articles in English, German, Spanish, Polish and Russian. The articles come from two curated lists of about 1,000 entries each, which were merged and deduplicated. Sentences, segmented with Stanza, were labeled by GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Each model ran three times, and only sentences on which all nine runs agreed entered the benchmark. Two multilingual BERT-style classifiers were then scored against these consensus labels. The scoring covered single sentences and whole documents, using two proposed aggregation strategies. The results show clear differences in label distributions, agreement and run-to-run stability across models and languages. The resource offers a dependable reference for low-subjectivity text. However, the authors caution that consensus labels are not true ground truth and may favour easier cases.

## Key points

- **Corpus size:** sentences per language after preprocessing were 21,991 (en), 12,715 (de), 17,021 (es), 10,707 (pl) and 12,739 (ru). Each sentence ID combines the page ID and the sentence index.
- **Annotation scheme:** the LLMs used four labels (positive, neutral, negative, n/a) with one shared prompt. Batches were sent as JSON lines of up to 25k tokens. The repeated runs measure intra-model stability as a stand-in for confidence.
- **Classifiers tested:**
  - **modBERT** (clapAI/modernBERT-base-multilingual-sentiment): about 150M parameters, trained on roughly 3.93M samples, reported F1 of 80.16. Polish is not among its training languages.
  - **mul-BERT** (tabularisai/multilingual-sentiment-analysis): DistilBERT-based, trained only on LLM-generated synthetic data, covers 23 languages including Polish.
- **Evaluation metric:** agreement is measured mainly with Cohen's κ. The authors warn that κ is unreliable on neutral-dominated data, where κ = 1.0 can simply mean every label was "neutral".
- **Stated limitations:** unanimous-consensus filtering trades coverage for reliability and drops borderline sentences. Other limits are the focus on lead sections only, five languages, a small set of models, and sensitivity to the prompt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Poznań University of Economics and Business](https://md.research.elamur.ai/free/en/universities/poznan-university-of-economics-and-business/) — University, Poznan, Poland · 3 works · 2 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Milena Stróżyna, Włodzimierz Lewoniewski and Izabela Czumałowska. 2026. Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{strozyna2026evaluating,
  title = {{Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark}},
  author = {Milena Stróżyna, Włodzimierz Lewoniewski and Izabela Czumałowska},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.63},
  url = {https://doi.org/10.18653/v1/2026.gem-main.63}
}
```

- [← previous Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks](https://md.research.elamur.ai/free/en/papers/maurya-2026-evaluating-llms-large-scale-graph/)
- [next → Evaluating Multimodal Large Language Model Narrative Interpretation through the Lens of Appraisal Theory](https://md.research.elamur.ai/free/en/papers/teotia-2026-evaluating-multimodal-large-language-model/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emmy Danny Ajik, Aminu Bashir Suleiman, and Muhammad Ibrahim. 2023. Enhancing user experience through sentiment analysis for katsina state transport agency: A textblob approach. FUDMA Journal of Sciences, 7(6):117–122. [find](https://scholar.google.com/scholar?q=Emmy+Danny+Ajik%2C+Aminu+Bashir+Suleiman%2C+and+Muhammad+Ibrahim%2E+2023%2E+Enhancing+user+experience+through+sentiment+analysis+for+katsina+state+transport+agency%3A+A+textblob+approach%2E+FUDMA+Journal+of+Scien)
2. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. [Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining](https://md.research.elamur.ai/free/en/papers/baccianella-2010-sentiwordnet-3-0-enhanced-lexical/). In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resour…
3. Hadis Bashiri and Hassan Naderi. 2024. Comprehensive review and comparative analysis of transformer models in sentiment analysis. Knowledge and Information Systems, 66(12):7305–7361. [find](https://scholar.google.com/scholar?q=Hadis+Bashiri+and+Hassan+Naderi%2E+2024%2E+Comprehensive+review+and+comparative+analysis+of+transformer+models+in+sentiment+analysis%2E+Knowledge+and+Information+Systems%2C+66%2812%29%3A7305%E2%80%937361%2E)
4. Anton Borg and Martin Boldt. 2020. Using vader sentiment and svm for predicting customer response sentiment. Expert Systems with Applications, 162:113746. [find](https://scholar.google.com/scholar?q=Anton+Borg+and+Martin+Boldt%2E+2020%2E+Using+vader+sentiment+and+svm+for+predicting+customer+response+sentiment%2E+Expert+Systems+with+Applications%2C+162%3A113746%2E)
5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
6. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
7. Himanshu Gautam, Abhishek Gaur, and Dharmendra Kumar Yadav. 2025. A survey on the impact of pre-trained language models in sentiment classification task. International Journal of Data Science and Analytics, 20(6):5197–5235. [find](https://scholar.google.com/scholar?q=Himanshu+Gautam%2C+Abhishek+Gaur%2C+and+Dharmendra+Kumar+Yadav%2E+2025%2E+A+survey+on+the+impact+of+pre%2Dtrained+language+models+in+sentiment+classification+task%2E+International+Journal+of+Data+Science+and+Anal)
8. K Puttaswamy Gowda, R Porwal, C Ramesh, SS Tiwari, K Srivastava, R Rambabu, and SG Rao. 2025. Transformers in sentiment analysis: A paradigm shift in deep learning research. J. Inf. Syst. Eng. Manag., 10(5):2468–4376. [find](https://scholar.google.com/scholar?q=K+Puttaswamy+Gowda%2C+R+Porwal%2C+C+Ramesh%2C+SS+Tiwari%2C+K+Srivastava%2C+R+Rambabu%2C+and+SG+Rao%2E+2025%2E+Transformers+in+sentiment+analysis%3A+A+paradigm+shift+in+deep+learning+research%2E+J%2E+Inf%2E+Syst%2E+Eng%2E+Manag%2E%2C)
9. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radliński, Konr… — [ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/)
10. Włodzimierz Lewoniewski, Milena Stróżyna, Izabela Czumałowska, and Elżbieta Lewańska. 2026a. Science fiction and fantasy in wikipedia: Exploring structural and semantic cues. arXiv preprint arXiv:2602.24229. — [arXiv:2602.24229](https://arxiv.org/abs/2602.24229) [find](https://scholar.google.com/scholar?q=W%C5%82odzimierz+Lewoniewski%2C+Milena+Str%C3%B3z%CC%87yna%2C+Izabela+Czuma%C5%82owska%2C+and+Elz%CC%87bieta+Lewan%CC%81ska%2E+2026a%2E+Science+fiction+and+fantasy+in+wikipedia%3A+Exploring+structural+and+semantic+cues%2E+arXiv+preprint+arXiv%3A2)
11. Włodzimierz Lewoniewski, Milena Stróżyna, Izabela Czumałowska, Aleksandra Wojewoda, and Krzysztof W˛ecel. 2026b. Cross-Topic Sentiment Analysis of Wikipedia Articles: A Comparative Study of AI Models. In Artificial Intelligence for Knowledge Acquisition and Management. AI4KAM 2025. IFIP Advances in… [find](https://scholar.google.com/scholar?q=W%C5%82odzimierz+Lewoniewski%2C+Milena+Str%C3%B3z%CC%87yna%2C+Izabela+Czuma%C5%82owska%2C+Aleksandra+Wojewoda%2C+and+Krzysztof+W%CB%9Becel%2E+2026b%2E+Cross%2DTopic+Sentiment+Analysis+of+Wikipedia+Articles%3A+A+Comparative+Study+of+AI+Models)
12. Yanying Mao, Qun Liu, and Yu Zhang. 2024. Sentiment analysis methods, applications, and challenges: A systematic literature review. Journal of King Saud University - Computer and Information Sciences, 36. [find](https://scholar.google.com/scholar?q=Yanying+Mao%2C+Qun+Liu%2C+and+Yu+Zhang%2E+2024%2E+Sentiment+analysis+methods%2C+applications%2C+and+challenges%3A+A+systematic+literature+review%2E+Journal+of+King+Saud+University+%2D+Computer+and+Information+Sciences%2C)
13. Sorin Adam Matei and Caius Dobrescu. 2011. Wikipedia’s "neutral point of view": Settling conflict through ambiguity. The Information Society, 27(1):40–51. [find](https://scholar.google.com/scholar?q=Sorin+Adam+Matei+and+Caius+Dobrescu%2E+2011%2E+Wikipedia%E2%80%99s+%22neutral+point+of+view%22%3A+Settling+conflict+through+ambiguity%2E+The+Information+Society%2C+27%281%29%3A40%E2%80%9351%2E)
14. Milena Stróżyna, Włodzimierz Lewoniewski, Izabela Czumałowska, and Aleksandra Wojewoda. 2025. Sentiment analysis of wikipedia articles about companies: A comparison of different models. In International Conference on Business Information Systems, pages 101–115. Springer. [find](https://scholar.google.com/scholar?q=Milena+Str%C3%B3z%CC%87yna%2C+W%C5%82odzimierz+Lewoniewski%2C+Izabela+Czuma%C5%82owska%2C+and+Aleksandra+Wojewoda%2E+2025%2E+Sentiment+analysis+of+wikipedia+articles+about+companies%3A+A+comparison+of+different+models%2E+In+Internatio)
15. Madhumita Sushil, Travis Zack, Divneet Mandair, Zhiwei Zheng, Ahmed Wali, Yan-Ning Yu, Yuwei Quan, Dmytro S. Lituiev, and Atul J. Butte. 2024. A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal… [find](https://scholar.google.com/scholar?q=Madhumita+Sushil%2C+Travis+Zack%2C+Divneet+Mandair%2C+Zhiwei+Zheng%2C+Ahmed+Wali%2C+Yan%2DNing+Yu%2C+Yuwei+Quan%2C+Dmytro+S%2E+Lituiev%2C+and+Atul+J%2E+Butte%2E+2024%2E+A+comparative+study+of+large+language+model%2Dbased+zero%2Dsh)

*Updated 1 November 2026*