--- title: "Ensemble Language Models for Multilingual Sentiment Analysis — Hasan 2024" description: "The paper looks at three-way tweet sentiment classification (positive, negative, neutral) in English and Arabic. 12 references, cited by 1 paper here." authors: ["Md Arid Hasan"] year: 2024 venue: "arXiv:2403.06060" source: "https://arxiv.org/abs/2403.06060" doi: "10.48550/arxiv.2403.06060" arxiv: "2403.06060" licence: "CC BY-NC-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/hasan-2024-ensemble-language-models-multilingual-sentiment/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hasan-2024-ensemble-language-models-multilingual-sentiment/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Ensemble Language Models for Multilingual Sentiment Analysis # Ensemble Language Models for Multilingual Sentiment Analysis **Authors:** [Md Arid Hasan](https://md.research.elamur.ai/free/en/authors/md-arid-hasan/) · **Year:** 2024 · **Venue:** arXiv:2403.06060 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2403.06060) · [PDF](https://arxiv.org/pdf/2403.06060) · [DOI 10.48550/arxiv.2403.06060](https://doi.org/10.48550/arxiv.2403.06060) · [arXiv 2403.06060](https://arxiv.org/abs/2403.06060) · [OpenAlex](https://openalex.org/W4392737165) **Citations:** 5 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 140,884 papers whose bibliographies we read ## Summary The paper looks at three-way tweet sentiment classification (positive, negative, neutral) in English and Arabic. Arabic is treated as the low-resource side. English data comes from SemEval-17 Task 4A plus earlier SemEval rounds. Arabic data merges SemEval-17 Arabic tweets with ASTD. Four pretrained transformers were fine-tuned separately for each language: AraBERTv02, RoBERTa base, multilingual BERT (the baseline) and XLM-RoBERTa base. Two ensemble architectures were then built. Each joins a language-specific encoder with m-BERT by concatenating their pooled outputs, passing them through a fusion layer and then a feed-forward classifier. One variant adds multi-head attention before the classifier. A majority-voting ensemble and training on the combined bilingual data were also tested. Macro-F1 is the main metric. Monolingual models clearly beat m-BERT, especially on Arabic. Majority voting gives the best English result. On Arabic, the plain AraBERTv02 model stays ahead of every ensemble. The study is a modest course-style project, but it gives a useful picture of how much language-specific pretraining matters compared with multilingual sharing. ## Key points - **Data:** - English train set: SemEval 2013–2016. The 2013/2014 test sets serve as dev, and the SemEval-17 test set is held out for evaluation. - Arabic: SemEval-17 subtasks A, B and D merged with ASTD (ASTD's "Objective" class removed), split 90/10 into train and validation. The official SemEval-17 subtask A test set is used for testing. - **Fine-tuning settings:** Adam with learning rate 2e-5, max length 256, batch size 16, 3 epochs, and BCE-with-logits loss. - **English macro-F1:** - m-BERT 67.06, RoBERTa 70.84, XLM-RoBERTa 69.13. - Majority voting is best at 71.03. - The feed-forward ensemble scores 68.59 when trained on English only and 69.88 when trained on combined data. - **Arabic macro-F1:** - m-BERT 53.08, AraBERTv02 69.78 (best), XLM-RoBERTa 63.74. - Ensembles land at 66.4–67.8. - The multi-head attention variant trained on combined data reaches only 66.42. - **Limitations stated by the author:** spot checks found mislabeled and biased annotations, no inter-annotator agreement was published, and no error analysis was done. ## Interesting ideas - Feeding a model both a language-specialised encoder and a broad shared encoder, then fusing their pooled vectors, lets specific and general knowledge complement each other without retraining either from scratch. - Simple majority voting over separately fine-tuned models matched or beat the learned fusion architectures, so a voting ensemble is a cheap and strong reference that any fancier combiner should have to beat. - Before merging two corpora, the author removed a label that existed in only one scheme so the class inventory matched, which is a minimal and transparent form of harmonisation. - Earlier editions' test sets are reused as the development split, so the newest official test set stays fully untouched for final scoring. - Hand-sampling a few gold items exposed annotation mistakes and bias, a reminder that gold labels themselves need auditing. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tree): gparse could concatenate a small domain-tuned encoder with a general one before the arc scorer, and ablations would show which source drives each attachment decision. - **I2** (learn): when witness parsers agree on silver trees, always report plain vote-agreement accuracy next to any weighted or learned combiner as a baseline control. - **I3** (tree): during UD dialect conversion, explicitly list relations that have no counterpart in the target scheme and drop or flag them rather than mapping them silently. - **I4** (learn): rotate older evaluation sets into tuning duty and keep a freshest held-out set that no repair rule or teacher tier has ever seen. - **I5** (learn): periodically sample gold treebank and answer-key items for manual review, and log disagreements as a separate error category instead of counting them as model failures. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Institutions - [University of New Brunswick](https://md.research.elamur.ai/free/en/universities/university-of-new-brunswick/) — University, Fredericton, Canada · 2 works · 9 citations ## Cite Cite as: Md Arid Hasan. 2024. Ensemble Language Models for Multilingual Sentiment Analysis. arXiv:2403.06060. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{hasan2024ensemble, title = {{Ensemble Language Models for Multilingual Sentiment Analysis}}, author = {Md Arid Hasan}, year = {2024}, doi = {10.48550/arxiv.2403.06060}, eprint = {2403.06060}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2403.06060} } ``` - [← previous Developing Arabic Grammar Materials Based on Contextual Learning at Islamic Boarding School in Indonesia](https://md.research.elamur.ai/free/en/papers/hamidah-2024-developing-arabic-grammar-materials-based/) - [next → HebDB: a Weakly Supervised Dataset for Hebrew Speech Processing](https://md.research.elamur.ai/free/en/papers/turetzky-2024-hebdb-weakly-supervised-dataset-hebrew/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 12 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/) 2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019. [↗](https://arxiv.org/abs/1911.02116) 3. Jingfeng Cui, Zhaoxia Wang, Seng-Beng Ho, and Erik Cambria. Survey on sentiment analysis: evolution of research methods and topics. Artificial Intelligence Review, pages 1–42, 2023. [find](https://scholar.google.com/scholar?q=Jingfeng+Cui%2C+Zhaoxia+Wang%2C+Seng%2DBeng+Ho%2C+and+Erik+Cambria%2E+Survey+on+sentiment+analysis%3A+evolution+of+research+methods+and+topics%2E+Artificial+Intelligence+Review%2C+pages+1%E2%80%9342%2C+2023%2E) 4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [↗](https://arxiv.org/abs/1810.04805) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) 5. Geetika Gautam and Divakar Yadav. Sentiment analysis of twitter data using machine learning approaches and semantic analysis. In 2014 Seventh International Conference on Contemporary Computing (IC3), pages 437–442, 2014. [our card](https://md.research.elamur.ai/free/en/papers/agarwal-2011-sentiment-analysis-twitter-data/) 6. Akshi Kumar and Victor Hugo C. Albuquerque. Sentiment analysis using xlm-r transformer and zero-shot transfer learning on resource-poor indian language. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 20(5), jun 2021. [find](https://scholar.google.com/scholar?q=Akshi+Kumar+and+Victor+Hugo+C%2E+Albuquerque%2E+Sentiment+analysis+using+xlm%2Dr+transformer+and+zero%2Dshot+transfer+learning+on+resource%2Dpoor+indian+language%2E+ACM+Trans%2E+Asian+Low%2DResour%2E+Lang%2E+Inf%2E+Process) 7. Wenxiong Liao, Bi Zeng, Xiuwen Yin, and Pengfei Wei. An improved aspect-category sentiment analysis model for text sentiment analysis based on roberta. Applied Intelligence, 51:3522–3533, 2021. [find](https://scholar.google.com/scholar?q=Wenxiong+Liao%2C+Bi+Zeng%2C+Xiuwen+Yin%2C+and+Pengfei+Wei%2E+An+improved+aspect%2Dcategory+sentiment+analysis+model+for+text+sentiment+analysis+based+on+roberta%2E+Applied+Intelligence%2C+51%3A3522%E2%80%933533%2C+2021%2E) 8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [↗](https://arxiv.org/abs/1907.11692) 9. George Manias, Argyro Mavrogiorgou, Athanasios Kiourtis, Chrysostomos Symvoulidis, and Dimosthenis Kyriazis. Multilingual text categorization and sentiment analysis: a comparative analysis of the utilization of multilingual approaches for classifying twitter data. Neural Computing and Applications, pages 1–17, 2023. [find](https://scholar.google.com/scholar?q=George+Manias%2C+Argyro+Mavrogiorgou%2C+Athanasios+Kiourtis%2C+Chrysostomos+Symvoulidis%2C+and+Dimosthenis+Kyriazis%2E+Multilingual+text+categorization+and+sentiment+analysis%3A+a+comparative+analysis+of+the+util) 10. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. Astd: Arabic sentiment tweets dataset. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2515–2519, 2015. [our card](https://md.research.elamur.ai/free/en/papers/nabil-2015-astd-arabic-sentiment-tweets-dataset/) 11. Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval-2017 task 4: Sentiment analysis in twitter. arXiv preprint arXiv:1912.00741, 2019. [↗](https://arxiv.org/abs/1912.00741) [our card](https://md.research.elamur.ai/free/en/papers/rosenthal-2017-semeval-2017-task-4-sentiment/) 12. Anshul Wadhawan. Arabert and farasa segmentation based approach for sarcasm and sentiment detection in arabic tweets. arXiv preprint arXiv:2103.01679, 2021. [↗](https://arxiv.org/abs/2103.01679) [our card](https://md.research.elamur.ai/free/en/papers/wadhawan-2021-arabert-farasa-segmentation-based-approach/)