---
title: "Fine-tuning Neural Language Models for Multidimensional"
description: "The paper builds benchmark classifiers for social opinion mining on online comments about Malta's… 31 references, cited by 1 paper here."
authors: ["Keith Cortis", "Kanishk Verma", "Brian Davis"]
year: 2021
venue: "Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)"
source: "https://aclanthology.org/2021.ranlp-1.36/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cortis-2021-fine-tuning-neural-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cortis-2021-fine-tuning-neural-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/9/) / Fine-tuning Neural Language Models for Multidimensional Opinion Mining of English-Maltese Social Data

# Fine-tuning Neural Language Models for Multidimensional Opinion Mining of English-Maltese Social Data

**Authors:** [Keith Cortis](https://md.research.elamur.ai/free/en/authors/keith-cortis/), Kanishk Verma and [Brian Davis](https://md.research.elamur.ai/free/en/authors/brian-davis/) · **Year:** 2021 · **Venue:** [Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)](https://md.research.elamur.ai/free/en/venues/ranlp/) ([RANLP 2021](https://md.research.elamur.ai/free/en/venues/ranlp/2021/)) · **Publisher:** [INCOMA Ltd.](https://md.research.elamur.ai/free/en/publishers/incoma-ltd/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.ranlp-1.36/) · [PDF](https://aclanthology.org/2021.ranlp-1.36.pdf) · [OpenAlex](https://openalex.org/W3214540015)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/cortis-2021-fine-tuning-neural-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper builds benchmark classifiers for social opinion mining on online comments about Malta's annual Government Budget. The data mixes English, the low-resourced Maltese, and code-switched Maltese-English. The authors use an existing annotated dataset of 6,387 posts from newswire comment sections and social media, covering the 2018–2020 budgets. Each post carries five opinion labels: subjectivity, sentiment polarity, emotion, sarcasm and irony. For every dimension they fine-tune four pretrained Transformer models with Hugging Face: BERT, DistilBERT, and two BERTweet variants. They do not rebalance the classes. Subjectivity and polarity are learned well, with the COVID-19 BERTweet model scoring best. Emotion is only moderate. For the rare sarcasm and irony classes, weighted F1 looks very high, but MCC shows that most models barely detect the positive class. The models are all English-based, yet the authors find their behaviour on the Maltese and mixed-language quarter of the data encouraging. The work offers early multilingual baselines for a language at risk of being left behind digitally.

## Key points
- **Dataset:** the Malta Budget social opinion dataset (Cortis and Davis, 2021a; Zenodo 10.5281/zenodo.4650232).
  - 6,387 posts in total.
  - About 74% are English, and 1,596 posts are Maltese or Maltese-English.
  - Sarcastic posts make up 2.77% and ironic posts 5.15%.
  - The emotion label uses 8 Plutchik-style classes, and fear accounts for only about 1%.
- **Setup:**
  - Random 70/20/10 train/validation/test split.
  - BERT-Base uncased, DistilBERT uncased, bertweet-base and bertweet-covid19-base-uncased.
  - Batch size 32, Adam with learning rate 2e-5, 4 epochs, max length 128, chosen by manual tuning.
- **Best results:**
  - Subjectivity: F1 0.94 and MCC 0.887, from BERTweet (covid19).
  - Polarity: F1 0.87 and MCC 0.781, also from BERTweet (covid19).
  - Emotion: F1 around 0.60 and MCC around 0.5, with BERT and DistilBERT best.
- **Sarcasm and irony:** F1 is 0.92–0.96, but MCC is no higher than 0.265 (sarcasm) and 0.240 (irony), both from DistilBERT. Both BERTweet models score an MCC of 0.
- **Training behaviour:** DistilBERT trained fastest. On the 8-class emotion task, BERTweet needed about 6 epochs to converge, beyond the usual 2–4.

## Interesting ideas
- Pairing weighted F1 with MCC exposed models that simply predict the majority class: they reached F1 0.96 on sarcasm while scoring MCC 0, so a chance-corrected metric acts as a built-in sanity check on skewed labels.
- Each post is annotated on several independent opinion layers at once, so subjectivity, polarity, emotion and figurative use can be studied separately and in combination.
- A small encoder pretrained on text of the same register (COVID tweets) beat general-domain BERT on noisy social comments, which suggests register match can matter more than corpus size.
- The authors did not use a fixed epoch budget; they read convergence from the training and validation loss per task and found that the many-class task needed more epochs.
- Class distributions are reported separately for each language subset, which shows how label skew differs between English and code-switched data.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Report MCC or a majority-class baseline next to accuracy for rare phenomena such as irony or rare relations, so that a degenerate model fails visibly.
- **I2** (frames): Treat irony, sarcasm and stance in crate prag as separate, individually explainable attributes of a speech act rather than one merged "sentiment" score.
- **I3** (tree): When picking or pretraining the encoder for gparse on user-generated text, prefer a register-matched model and compare it against a general one.
- **I4** (learn): Set stopping criteria per task from validation curves instead of a global epoch count, especially for tag sets with many rare labels.
- **I5** (learn): Slice silver-data statistics and evaluation by register or source, so that hidden distribution shifts are visible before training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Dublin City University](https://md.research.elamur.ai/free/en/universities/dublin-city-university/) — University, Dublin, Ireland · 72 works · 9,746 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Keith Cortis, Kanishk Verma and Brian Davis. 2021. Fine-tuning Neural Language Models for Multidimensional Opinion Mining of English-Maltese Social Data. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cortis2021fine,
  title = {{Fine-tuning Neural Language Models for Multidimensional Opinion Mining of English-Maltese Social Data}},
  author = {Keith Cortis, Kanishk Verma and Brian Davis},
  booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)},
  year = {2021},
  doi = {10.26615/978-954-452-072-4\_036},
  url = {https://doi.org/10.26615/978-954-452-072-4_036}
}
```

- [← previous False resultatives: The interaction of agreement and creation in Northern Galilee Levantine Arabic](https://md.research.elamur.ai/free/en/papers/rapoport-2021-false-resultatives-interaction-agreement-creation/)
- [next → Identifying the Old English exponent for the semantic prime LIVE](https://md.research.elamur.ai/free/en/papers/mendaza-2021-identifying-old-english-exponent-semantic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/9/)

## References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aggeliki Androutsopoulou, Nikos Karacapilidis, Euripidis Loukis, and Yannis Charalabidis. 2019. Transforming the communication between citizens and government through ai-guided chatbots. Government Information Quarterly, 36(2):358–367. [find](https://scholar.google.com/scholar?q=Aggeliki+Androutsopoulou%2C+Nikos+Karacapilidis%2C+Euripidis+Loukis%2C+and+Yannis+Charalabidis%2E+2019%2E+Transforming+the+communication+between+citizens+and+government+through+ai%2Dguided+chatbots%2E+Government+In)
2. Nastaran Babanejad, Heidar Davoudi, Aijun An, and Manos Papagelis. 2020. [Affective and contextual embedding for sarcasm detection](https://md.research.elamur.ai/free/en/papers/babanejad-2020-affective-contextual-embedding-sarcasm-detection/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 225–243, Barcelona, Spain (Online). International Committee on Computational Linguistics.
3. Silvia Cateni, Valentina Colla, and Marco Vannucci. 2014. A method for resampling imbalanced datasets in binary classification tasks for real-world problems. Neurocomputing, 135:32–41. [find](https://scholar.google.com/scholar?q=Silvia+Cateni%2C+Valentina+Colla%2C+and+Marco+Vannucci%2E+2014%2E+A+method+for+resampling+imbalanced+datasets+in+binary+classification+tasks+for+real%2Dworld+problems%2E+Neurocomputing%2C+135%3A32%E2%80%9341%2E)
4. Keith Cortis and Brian Davis. 2021a. A dataset of multidimensional and multilingual social opinions for malta’s annual government budget. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 971–981. [find](https://scholar.google.com/scholar?q=Keith+Cortis+and+Brian+Davis%2E+2021a%2E+A+dataset+of+multidimensional+and+multilingual+social+opinions+for+malta%E2%80%99s+annual+government+budget%2E+In+Proceedings+of+the+International+AAAI+Conference+on+Web+and)
5. Keith Cortis and Brian Davis. 2021b. Over a decade of social opinion mining: a systematic review. Artificial intelligence review, pages 1–93. [find](https://scholar.google.com/scholar?q=Keith+Cortis+and+Brian+Davis%2E+2021b%2E+Over+a+decade+of+social+opinion+mining%3A+a+systematic+review%2E+Artificial+intelligence+review%2C+pages+1%E2%80%9393%2E)
6. Danilo Croce, Giuseppe Castellucci, and Roberto Basili. 2020. GAN-BERT: Generative adversarial learning for robust text classification with a bunch of labeled examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2114–2119, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.191)
7. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dorottya+Demszky%2C+Dana+Movshovitz%2DAttias%2C+Jeongwoo+Ko%2C+Alan+Cowen%2C+Gaurav+Nemade%2C+and+Sujith+Ravi%2E+2020%2E+GoEmotions%3A+A+dataset+of+fine%2Dgrained+emotions%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
9. Hongliang Fei and Ping Li. 2020. Cross-lingual unsupervised sentiment classification with multi-view transfer learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5759–5771. [find](https://scholar.google.com/scholar?q=Hongliang+Fei+and+Ping+Li%2E+2020%2E+Cross%2Dlingual+unsupervised+sentiment+classification+with+multi%2Dview+transfer+learning%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+L)
10. Digital Economy Parliamentary Secretariat for Financial Services and Office of the Prime Minister Innovation. 2019. Malta: The ultimate ai launchpad - a strategy and vision for artificial intelligence in malta 2030. [https://malta.ai/wp-content/uploads/2019/10/Malta_The_Ultimate_](https://malta.ai/wp-content/uploads/2019/10/Malta_The_Ultimate_) AI_Launchpad_vFinal.pdf . Date Published: 2019-10-02. [↗](https://malta.ai/wp-content/uploads/2019/10/Malta_The_Ultimate_)
11. M Geetha, Pratap Singha, and Sumedha Sinha. 2017. Relationship between customer sentiment and online customer ratings for hotels-an empirical analysis. Tourism Management, 61:43–54. [find](https://scholar.google.com/scholar?q=M+Geetha%2C+Pratap+Singha%2C+and+Sumedha+Sinha%2E+2017%2E+Relationship+between+customer+sentiment+and+online+customer+ratings+for+hotels%2Dan+empirical+analysis%2E+Tourism+Management%2C+61%3A43%E2%80%9354%2E)
12. Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. 2020. Fine-tuning BERT for low-resource natural language understanding via active learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1158–1171, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daniel+Grie%C3%9Fhaber%2C+Johannes+Maucher%2C+and+Ngoc+Thang+Vu%2E+2020%2E+Fine%2Dtuning+BERT+for+low%2Dresource+natural+language+understanding+via+active+learning%2E+In+Proceedings+of+the+28th+International+Conference+)
13. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
14. Fajri Koto, Afshin Rahimi, Jey Han Lau, and Timothy Baldwin. 2020. IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In Proceedings of the 28th International Conference on Computational Linguistics, pages 757–770, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Afshin+Rahimi%2C+Jey+Han+Lau%2C+and+Timothy+Baldwin%2E+2020%2E+IndoLEM+and+IndoBERT%3A+A+benchmark+dataset+and+pre%2Dtrained+language+model+for+Indonesian+NLP%2E+In+Proceedings+of+the+28th+International)
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
16. Victor Makarenkov and Lior Rokach. 2020. Lessons learned from applying off-the-shelf bert: There is no silverbullet. arXiv preprint arXiv:2009.07238. [↗](https://arxiv.org/abs/2009.07238)
17. Ajinkya More. 2016. Survey of resampling techniques for improving classification performance in unbalanced datasets. arXiv preprint arXiv:1608.06048. [↗](https://arxiv.org/abs/1608.06048)
18. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. [BERTweet: A pre-trained language model for English Tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14.
19. Emily Öhman, Marc Pàmies, Kaisla Kajava, and Jörg Tiedemann. 2020. XED: A multilingual dataset for sentiment analysis and emotion detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6542–6552, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emily+O%CC%88hman%2C+Marc+Pa%CC%80mies%2C+Kaisla+Kajava%2C+and+Jo%CC%88rg+Tiedemann%2E+2020%2E+XED%3A+A+multilingual+dataset+for+sentiment+analysis+and+emotion+detection%2E+In+Proceedings+of+the+28th+International+Conference+on+C)
20. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. [Semeval-2017 task 4: Sentiment analysis in twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2017-semeval-2017-task-4-sentiment/). In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502– 518.
21. Mike Rosner, Jan Joachimsen, Georg Rehm, and Hans Uszkoreit. 2012. The Maltese language in the digital age. Springer. [find](https://scholar.google.com/scholar?q=Mike+Rosner%2C+Jan+Joachimsen%2C+Georg+Rehm%2C+and+Hans+Uszkoreit%2E+2012%2E+The+Maltese+language+in+the+digital+age%2E+Springer%2E)
22. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). arXiv preprint arXiv:1910.01108. [↗](https://arxiv.org/abs/1910.01108)
23. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d13-1170)
24. Emma Strubell, Ananya Ganesh, and Andrew Mc-Callum. 2019. [Energy and policy considerations for deep learning in nlp](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). arXiv preprint arXiv:1906.02243. [↗](https://arxiv.org/abs/1906.02243)
25. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. [Semeval-2018 task 3: Irony detection in english tweets](https://md.research.elamur.ai/free/en/papers/hee-2018-semeval-2018-task-3-irony/). In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50.
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). arXiv preprint arXiv:1706.03762. [↗](https://arxiv.org/abs/1706.03762)
27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771. [↗](https://arxiv.org/abs/1910.03771)
28. Patrick Xia, Shijie Wu, and Benjamin Van Durme. 2020. Which *BERT? A survey organizing contextualized encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7516–7533, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Patrick+Xia%2C+Shijie+Wu%2C+and+Benjamin+Van+Durme%2E+2020%2E+Which+%2ABERT%3F+A+survey+organizing+contextualized+encoders%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
29. Ali Yadollahi, Ameneh Gholipour Shahraki, and Osmar R Zaiane. 2017. Current state of text sentiment analysis from opinion to emotion mining. ACM Computing Surveys (CSUR), 50(2):1–33. [find](https://scholar.google.com/scholar?q=Ali+Yadollahi%2C+Ameneh+Gholipour+Shahraki%2C+and+Osmar+R+Zaiane%2E+2017%2E+Current+state+of+text+sentiment+analysis+from+opinion+to+emotion+mining%2E+ACM+Computing+Surveys+%28CSUR%29%2C+50%282%29%3A1%E2%80%9333%2E)
30. Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele, and Chris Biemann. 2020. [Exploring amharic sentiment analysis from social media texts: Building annotation tools and classification models](https://md.research.elamur.ai/free/en/papers/yimam-2020-exploring-amharic-sentiment-analysis-social/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 1048–1060.
31. Yabing Zhao, Xun Xu, and Mingshu Wang. 2019. Predicting overall customer satisfaction: Big data evidence from hotel online textual reviews. International Journal of Hospitality Management, 76:111– 121. [find](https://scholar.google.com/scholar?q=Yabing+Zhao%2C+Xun+Xu%2C+and+Mingshu+Wang%2E+2019%2E+Predicting+overall+customer+satisfaction%3A+Big+data+evidence+from+hotel+online+textual+reviews%2E+International+Journal+of+Hospitality+Management%2C+76%3A111%E2%80%93+121)

*Updated 2 November 2026*