---
title: "LLMs vs. Traditional Sentiment Tools in Psychology: An"
description: "The paper asks whether recent Dutch-adapted LLMs estimate emotional valence in everyday Flemish… 40 references, cited by 1 paper here."
authors: ["Ratna Kandala", "Katie Hoemann"]
year: 2025
venue: "arXiv:2511.07641"
source: "https://arxiv.org/abs/2511.07641"
doi: "10.48550/arxiv.2511.07641"
arxiv: "2511.07641"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kandala-2025-llms-vs-traditional-sentiment-tools/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kandala-2025-llms-vs-traditional-sentiment-tools/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/) / LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives

# LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives

**Authors:** [Ratna Kandala](https://md.research.elamur.ai/free/en/authors/ratna-kandala/) and [Katie Hoemann](https://md.research.elamur.ai/free/en/authors/katie-hoemann/) · **Year:** 2025 · **Venue:** arXiv:2511.07641 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2511.07641) · [PDF](https://arxiv.org/pdf/2511.07641) · [DOI 10.48550/arxiv.2511.07641](https://doi.org/10.48550/arxiv.2511.07641) · [arXiv 2511.07641](https://arxiv.org/abs/2511.07641) · [OpenAlex](https://openalex.org/W4416189324)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether recent Dutch-adapted LLMs estimate emotional valence in everyday Flemish writing better than classic lexicon tools used in psychology. The authors used an ambulatory-assessment corpus of 24,854 free-text entries from 102 participants in Belgium, gathered over 70 days. Four times a day, participants described what was happening and how they felt, and rated their own valence on a –50 to +50 scale. Those self-ratings, rather than annotator labels, served as ground truth. Three open-weight Dutch LLMs, given a single English prompt asking for a 1–7 rating, were compared with LIWC and Pattern.nl using Pearson and polyserial correlations. The lexicon tools scored almost every text. The LLMs often failed to return a usable rating. Pattern.nl was the most reliable tool overall. On the texts they did score, ChocoLlama and GEITje correlated somewhat better with self-reports. The work matters for low-resource variants because coverage and domain fit can outweigh model sophistication when tools are applied to real, colloquial narratives.

## Key points
- **Data:** 24,854 Dutch/Flemish entries from 102 participants (ages 18–65) over 70 days. Each entry was a short text or a 1-minute voice note, paired with a self-rated valence from –50 to +50.
- **Models and tools:** ChocoLlama-8B-Instruct (Llama-3-8B, LoRA), GEITje-7B-ultra (Mistral 7B, full pretraining), and Reynaerde-7B-chat (Llama-2-7B, QLoRA) were compared with LIWC (Dutch dictionary of 6,614 words) and Pattern.nl (3,304 lemmas, mostly adjectives).
- **Coverage:**
  - LIWC and Pattern.nl: 99.9%
  - ChocoLlama: 69.9%
  - GEITje: 38.0%
  - Reynaerde: 1.8%
- **Correlations with self-reports (Pearson / polyserial):**
  - Pattern.nl: 0.31 / 0.31
  - LIWC PosEmo: 0.21 / 0.23
  - LIWC NegEmo: –0.23 / –0.23
  - ChocoLlama: 0.35 / 0.40 on its subset, significantly above Pattern there (t = 4.02)
  - GEITje: 0.35 / 0.44, not significantly different from Pattern
  - Reynaerde: 0.18 / 0.24
- **Prompt variations:** Switching between Dutch and English prompts made no notable difference, and neither did zero-shot versus few-shot prompting. The authors attribute LLM weakness to a mismatch between formal training corpora and casual first-person narratives.

## Interesting ideas
- Using the writer's own rating as the gold label avoids annotator guesswork about inner states, which suits any task where the author is the true authority on meaning.
- Reporting coverage next to accuracy, and comparing systems on the same subset, shows that a model which silently fails on many inputs can look stronger than it is.
- Pattern.nl beat raw word counting by scoring chunks and applying negation and intensifier rules before aggregating, so even simple compositional structure adds signal.
- A lexicon built from product reviews ends up treating out-of-domain emotional words as neutral, so the origin of a resource quietly shapes its errors.
- Forcing an LLM into a strict output format can fail massively, as Reynaerde's 1.8% usable answers show, so output parsing is itself a failure point to measure.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Where possible, Mova could score its emotion or attitude readings against author-supplied labels, keeping these separate from judgments inferred by outside annotators or LLM teachers.
- **I2** (learn): Every evaluation, especially of LLM teacher tiers, should log abstentions and malformed outputs and measure scores on the intersection of inputs, so selection bias cannot inflate the results.
- **I3** (frames): Polarity could be computed over the dependency tree, flipping under negation and scaling under degree modifiers, giving an explainable attitude value per frame instead of a bag-of-words tally.
- **I4** (world): Compiled value and sentiment knowledge should record its source domain, letting Mova flag when a text's vocabulary falls outside what the resource was built on.
- **I5** (learn): LLM teacher outputs should be passed through a validated parser, with per-model failure rates tracked so that unreliable teachers are down-weighted rather than trusted.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 306 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works

## Cite

Cite as: Ratna Kandala and Katie Hoemann. 2025. LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives. arXiv:2511.07641.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kandala2025llms,
  title = {{LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives}},
  author = {Ratna Kandala and Katie Hoemann},
  year = {2025},
  doi = {10.48550/arxiv.2511.07641},
  eprint = {2511.07641},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.07641}
}
```

- [← previous LIS at BAREC Shared Task 2025: Multi-Scale Curriculum Learning for Arabic Sentence-Level Readability Assessment Using Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/djoudi-2025-lis-barec-shared-task-2025/)
- [next → Lost in Variation: An Unsupervised Methodology for Mining Lexico-syntactic Patterns in Middle Arabic Texts](https://md.research.elamur.ai/free/en/papers/bezancon-2025-lost-variation-unsupervised-methodology-mining/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/)

## References

40 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexandridis, G., Varlamis, I., Korovesis, K., Caridakis, G., and Tsantilas, P. (2021). A survey on sentiment analysis and opinion mining in Greek social media. Information, 12:331. [↗](https://doi.org/10.3390/info12080331)
2. Ali, H., Hashmi, E., Yildirim, S. Y., and Shaikh, S. (2024). Analyzing Amazon products sentiment: a comparative study of machine and deep learning, and transformer-based techniques. Electronics, 13(7):1305. [↗](https://doi.org/10.3390/electronics13071305)
3. Augustyniak, Ł., Woźniak, S., Gruza, M., Gramacki, P., Rajda, K., Morzy, M., and Kajdanowicz, T. (2023). Massively multilingual corpus of sentiment datasets and multi-faceted sentiment classification benchmark. arXiv preprint arXiv:2306.07902. [↗](https://arxiv.org/abs/2306.07902)
4. Blitzer, J., Dredze, M., and Pereira, F. (2007). Biographies, Bollywood, boom-boxes: domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pp. 440–447. [find](https://scholar.google.com/scholar?q=Blitzer%2C+J%2E%2C+Dredze%2C+M%2E%2C+and+Pereira%2C+F%2E+%282007%29%2E+Biographies%2C+Bollywood%2C+boom%2Dboxes%3A+domain+adaptation+for+sentiment+classification%2E+In+Proceedings+of+the+45th+Annual+Meeting+of+the+Association+for+Co)
5. Boot, P., Zijlstra, H., and Geenen, R. (2017). The Dutch translation of the Linguistic Inquiry and Word Count (LIWC) 2007 dictionary. Dutch Journal of Applied Linguistics, 6(1):65–76. [↗](https://doi.org/10.1075/dujal.6.1.04boo)
6. Bordia, S. and Bowman, S. R. (2019). [Identifying and Reducing Gender Bias in Word-Level Language Models](https://md.research.elamur.ai/free/en/papers/bordia-2019-identifying-reducing-gender-bias-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 7–15. Association for Computational Linguistics. https://aclanthology.org/N19-3002/. [↗](https://aclanthology.org/N19-3002/)
7. Chun, J. (2021). SentimentArcs: a novel method for self-supervised sentiment analysis of time series shows SOTA transformers can struggle finding narrative arcs. arXiv preprint arXiv:2110.09454. [↗](https://arxiv.org/abs/2110.09454)
8. Dashtipour, K., Gogate, M., Adeel, A., Larijani, H., and Hussain, A. (2021). Sentiment analysis of Persian movie reviews using deep learning. Entropy, 23:596. [↗](https://doi.org/10.3390/e23050596)
9. De Smedt, T. and Daelemans, W. (2012). Pattern for Python. Journal of Machine Learning Research, 13(1):2063–2067. [↗](https://doi.org/10.5555/2188385.2343710)
10. Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., Jones, M., Krettek-Cobb, D., Lai, L., Mitchell, N. J., Ong, D. C., Dweck, C. S., Gross, J. J., and Pennebaker, J. W. (2023). Using large language models in psychology. Nature Reviews Psychology, 2(11):688–701. [↗](https://doi.org/10.1038/s44159-023-00241-5)
11. Fiok, K., Karwowski, W., Gutierrez, E., and Wilamowski, M. (2021). Analysis of sentiment in tweets addressed to a single domain specific Twitter account: comparison of model performance and explainability of predictions. Expert Systems with Applications, 186:115771. [↗](https://doi.org/10.1016/j.eswa.2021.115771)
12. Frisina, P. G., Borod, J. C., and Lepore, S. J. (2004). A meta-analysis of the effects of written emotional disclosure on the health outcomes of clinical populations. The Journal of Nervous and Mental Disease, 192(9):629–634. [↗](https://doi.org/10.1097/01.nmd.0000138317.30764.63)
13. Gatti, L., and van Stegeren, J. (2020). Improving Dutch sentiment analysis in Pattern. Computational Linguistics in the Netherlands Journal, 10:73–89. [find](https://scholar.google.com/scholar?q=Gatti%2C+L%2E%2C+and+van+Stegeren%2C+J%2E+%282020%29%2E+Improving+Dutch+sentiment+analysis+in+Pattern%2E+Computational+Linguistics+in+the+Netherlands+Journal%2C+10%3A73%E2%80%9389%2E)
14. Hutto, C. J. and Gilbert, E. (2014). VADER: a parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the International AAAI Conference on Web and Social Media, 8(1):216–225. [find](https://scholar.google.com/scholar?q=Hutto%2C+C%2E+J%2E+and+Gilbert%2C+E%2E+%282014%29%2E+VADER%3A+a+parsimonious+rule%2Dbased+model+for+sentiment+analysis+of+social+media+text%2E+In+Proceedings+of+the+International+AAAI+Conference+on+Web+and+Social+Media%2C+8%28)
15. Li, Z., Shi, Y., Liu, Z., Yang, F., Payani, A., Liu, N., and Du, M. (2024). [Quantifying multilingual performance of large language models across languages](https://md.research.elamur.ai/free/en/papers/li-2024-language-ranker-metric-quantifying-llm/). arXiv preprint arXiv:2404.11553. [↗](https://arxiv.org/abs/2404.11553)
16. Ligthart, A., Catal, C., and Tekinerdogan, B. (2021). Systematic reviews in sentiment analysis: a tertiary study. Artificial Intelligence Review, 54:4997–5053. [↗](https://doi.org/10.1007/s10462-021-09973-3)
17. Liu, Z., Zhang, T., Yang, K., Thompson, P., Yu, Z., and Ananiadou, S. (2023). Emotion detection for misinformation: a review. arXiv preprint arXiv:2311.00671. [↗](https://arxiv.org/abs/2311.00671)
18. Liu, Z., Yang, K., Xie, Q., Zhang, T., and Ananiadou, S. (2024). EmoLLMs: a series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5487–5496. [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Yang%2C+K%2E%2C+Xie%2C+Q%2E%2C+Zhang%2C+T%2E%2C+and+Ananiadou%2C+S%2E+%282024%29%2E+EmoLLMs%3A+a+series+of+emotional+large+language+models+and+annotation+tools+for+comprehensive+affective+analysis%2E+In+Proceedings+of+the+3)
19. Martínez, G., Molero, J. D., González, S., Conde, J., Brysbaert, M., and Reviriego, P. (2025). Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57:5. https://doi.org/10.3758/s13428-024-02515-z [↗](https://doi.org/10.3758/s13428-024-02515-z)
20. Meeus, M., Rathé, A., Rémy, F., Delobelle, P., Decorte, J.-J., and Demeester, T. (2024). ChocoLlama: lessons learned from teaching llamas Dutch. arXiv preprint arXiv:2412.07633. [↗](https://arxiv.org/abs/2412.07633)
21. Mishra, K., Priya, P., Burja, M., and Ekbal, A. (2023). e-THERAPIST: I suggest you to cultivate a mindset of positivity and nurture uplifting thoughts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13952–13967. [find](https://scholar.google.com/scholar?q=Mishra%2C+K%2E%2C+Priya%2C+P%2E%2C+Burja%2C+M%2E%2C+and+Ekbal%2C+A%2E+%282023%29%2E+e%2DTHERAPIST%3A+I+suggest+you+to+cultivate+a+mindset+of+positivity+and+nurture+uplifting+thoughts%2E+In+Proceedings+of+the+2023+Conference+on+Empiric)
22. Mohammad, S. M. and Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465. [↗](https://doi.org/10.1111/j.1467-8640.2012.00460.x)
23. Muhammad, S., Abdulmumin, I., Ayele, A., Ousidhoum, N., Adelani, D., Yimam, S., Ahmad, I., Beloucif, M., Mohammad, S., Ruder, S., Hourrane, O., Jorge, A., Brazdil, P., Ali, F., David, D., Osei, S., Shehu-Bello, B., Lawan, F., Gwadabe, T., Rutunda, S., Belay, T., Messelle, W., Balcha, H., Chala, S., Gebremichael, H., and Opoku, B. (2023). [AfriSenti: a Twitter sentiment analysis benchmark for African languages](https://md.research.elamur.ai/free/en/papers/muhammad-2023-afrisenti-twitter-sentiment-analysis-benchmark/). In Proceedings of EMNLP 2023. [↗](https://doi.org/10.48550/arxiv.2302.08956)
24. Obradovich, N., Khalsa, S. S., Khan, W. U., et al. (2024). Opportunities and risks of large language models in psychiatry. NPP—Digital Psychiatry Neuroscience, 2:8. [↗](https://doi.org/10.1038/s44277-024-00010-z)
25. Pataranutaporn, P., Liu, R., Finn, E., et al. (2023). Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence, 5:1076–1086. [↗](https://doi.org/10.1038/s42256-023-00720-7)
26. Pennebaker, J. W. (1997). Writing about emotional experiences as a therapeutic process. Psychological Science, 8(3):162–166. [↗](https://doi.org/10.1111/j.1467-9280.1997.tb00403.x)
27. Pennebaker, J. W., Francis, M. E., and Booth, R. J. (2001). Linguistic Inquiry and Word Count: LIWC 2001. Lawrence Erlbaum Associates, Mahwah, NJ. [find](https://scholar.google.com/scholar?q=Pennebaker%2C+J%2E+W%2E%2C+Francis%2C+M%2E+E%2E%2C+and+Booth%2C+R%2E+J%2E+%282001%29%2E+Linguistic+Inquiry+and+Word+Count%3A+LIWC+2001%2E+Lawrence+Erlbaum+Associates%2C+Mahwah%2C+NJ%2E)
28. Pennebaker, J. W., Booth, R. J., and Francis, M. E. (2007). Linguistic Inquiry and Word Count: LIWC2007: operator’s manual. Austin, TX: LIWC.net. [find](https://scholar.google.com/scholar?q=Pennebaker%2C+J%2E+W%2E%2C+Booth%2C+R%2E+J%2E%2C+and+Francis%2C+M%2E+E%2E+%282007%29%2E+Linguistic+Inquiry+and+Word+Count%3A+LIWC2007%3A+operator%E2%80%99s+manual%2E+Austin%2C+TX%3A+LIWC%2Enet%2E)
29. Pennebaker, J. W., Booth, R. J., Boyd, R. L., and Francis, M. E. (2015). Linguistic Inquiry and Word Count: LIWC2015. Austin, TX: Pennebaker Conglomerates. [https://www.liwc.net](https://www.liwc.net) [↗](https://www.liwc.net)
30. Qiu, J., Lam, K., Li, G., et al. (2024). LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6:1418–1420. [↗](https://doi.org/10.1038/s42256-024-00944-1)
31. Radaideh, M. I., Hwang, O. H., and Radaideh, M. I. (2024). Fairness and social bias quantification in large language models for sentiment analysis. SSRN. [↗](https://doi.org/10.1016/j.knosys.2025.113569)
32. ReBatch. (2024). Reynaerde-7B-Chat. Hugging Face. [find](https://scholar.google.com/scholar?q=ReBatch%2E+%282024%29%2E+Reynaerde%2D7B%2DChat%2E+Hugging+Face%2E)
33. Reusens, M., Reusens, M., Callens, M., Vanden Broucke, S., and Baesens, B. (2022). Benchmark study for Flemish Twitter sentiment analysis. SSRN. https://ssrn.com/abstract=4096559 [↗](https://doi.org/10.2139/ssrn.4096559)
34. Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., and Rand, D. G. (2014). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4):1064–1082. https://doi.org/10.1111/ajps.12103 [↗](https://doi.org/10.1111/ajps.12103)
35. Shteynberg, G., Halpern, J., Sadovnik, A., et al. (2024). Does it matter if empathic AI has no empathy? Nature Machine Intelligence, 6:496–497. [↗](https://doi.org/10.1038/s42256-024-00841-7)
36. Tausczik, Y. R. and Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1):24–54. [↗](https://doi.org/10.1177/0261927x09351676)
37. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. (2023). [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288. https://arxiv.org/abs/2307.09288 [↗](https://arxiv.org/abs/2307.09288)
38. Vanroy, B. (2024). GEITje 7B Ultra: a conversational model for Dutch. arXiv preprint arXiv:2412.04092. [↗](https://arxiv.org/abs/2412.04092)
39. Varghese, C., Harrison, E. M., O’Grady, G., et al. (2024). Artificial intelligence in surgery. Nature Medicine, 30:1257–1268. [↗](https://doi.org/10.1038/s41591-024-02970-3)
40. Zhang, W., Deng, Y., Liu, B., Pan, S., and Bing, L. (2024). [Sentiment analysis in the era of large language models: a reality check](https://md.research.elamur.ai/free/en/papers/zhang-2023-sentiment-analysis-era-large-language/). In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3881–3906.

*Updated 1 November 2026*