--- title: "Benchmarking transfer learning approaches for sentiment" description: "The paper describes the authors' system for the country-level sentiment subtask of the third NADI… 13 references, cited by 2 papers here." authors: ["Emna Fsih", "Sameh Kchaou", "Rahma Boujelbane", "Lamia Hadrich-Belguith"] year: 2022 venue: "Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)" source: "https://aclanthology.org/2022.wanlp-1.44/" doi: "10.18653/v1/2022.wanlp-1.44" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/fsih-2022-benchmarking-transfer-learning-approaches-sentiment/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fsih-2022-benchmarking-transfer-learning-approaches-sentiment/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/7/) / Benchmarking transfer learning approaches for sentiment analysis of Arabic dialect # Benchmarking transfer learning approaches for sentiment analysis of Arabic dialect **Authors:** Emna Fsih, [Sameh Kchaou](https://md.research.elamur.ai/free/en/authors/sameh-kchaou/), [Rahma Boujelbane](https://md.research.elamur.ai/free/en/authors/rahma-boujelbane/) and [Lamia Hadrich-Belguith](https://md.research.elamur.ai/free/en/authors/lamia-hadrich-belguith/) · **Year:** 2022 · **Venue:** [Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2022](https://md.research.elamur.ai/free/en/venues/wanlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.wanlp-1.44/) · [PDF](https://aclanthology.org/2022.wanlp-1.44.pdf) · [DOI 10.18653/v1/2022.wanlp-1.44](https://doi.org/10.18653/v1/2022.wanlp-1.44) · [OpenAlex](https://openalex.org/W4385573265) **Citations:** 7 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/fsih-2022-benchmarking-transfer-learning-approaches-sentiment/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes the authors' system for the country-level sentiment subtask of the third NADI shared task. The task is to classify dialectal Arabic tweets as positive, negative or neutral when only a small labelled set is available. The authors clean the tweets and normalise Arabic orthography with CAMeL tools. They then enlarge the training data in two ways. One uses masked-language-model insertion and substitution via NLPAug. The other adds or removes emojis according to sentiment. They fine-tune three transfer-learning models: a multi-dialect Arabic BERT (M1_Bashar), CAMeLBERT (M2_CAMeL) and SetFit, a few-shot Sentence Transformer approach (M3_SetFit). M1 scores best on DEV but collapses on TEST. The emoji-augmented CAMeLBERT and the SetFit baseline give the strongest TEST scores, at roughly 43% and 42% macro F1-PN. Error analysis shows that the neutral class is almost never recognised. It also uncovers mislabelled and inconsistently labelled tweets in the official data. The study is useful as a candid picture of how unreliable development scores can be with small, noisy dialect corpora. ## Key points - **Data:** the NADI-2022 sentiment data has 1500 TRAIN, 500 DEV and 3000 TEST tweets. Neutral is the smallest class, with 340 neutral training tweets against about 580 for each polar class. - **Metric and baselines:** results are reported as macro F1-PN. The baselines on DEV/TEST are: - M1_Bashar: 66.72% / 39.69% - M2_CAMeL: 47.85% / 41.72% - M3_SetFit: 43.23% / 42.33% - **Contextual augmentation:** insertion or substitution with multi-dialect-bert-base-arabic through NLPAug added 3000 sentences. - **Emoji augmentation:** attaching class-specific emojis added 1000 sentences. - **Best augmented results:** - M2_aug_emoj reached 54.00% DEV and 43.11% TEST, the top TEST score overall. - M1_aug_emoj reached 65.33% DEV but only 39.38% TEST. - Augmentation gave only small and inconsistent changes. - **Error analysis:** the SetFit confusion matrix shows roughly 50% correct predictions on the polar classes and near-failure on neutral. Several "errors" turned out to be gold-label mistakes. Two near-identical tweets, differing only by a typo, carried different labels. ## Interesting ideas - Reporting DEV and TEST side by side exposed a huge gap for the model that looked best on DEV, a warning against choosing models on a small dev split. - Emoji augmentation first manually selects emojis that appear only in one class, then attaches them to that class's examples that lack such cues. This makes the cue-label link explicit rather than random. - Inspecting the confusion matrix led the authors back to the gold data, where they found wrong labels and near-duplicate tweets annotated differently. - Masked-LM insertion and substitution produces many slightly altered variants of each sentence from a small corpus. - Folding confusable orthographic variants (the HAMZA forms) into one representative character before modelling reduces vocabulary sparsity on noisy user text. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Always report a held-out set the system never tuned on next to the tuning set, and treat a large gap as a red flag before trusting a new parser or repair rule. - **I2** (learn): Silver-data augmentation could inject only cues verified to occur exclusively with one analysis, and a negative control could check that the model is not just learning the injected cue. - **I3** (tree): Grouping treebank sentences that differ by a character or token and comparing their UD annotations could cheaply surface inconsistent gold trees before training gparse or the perceptron parser. - **I4** (learn): Such perturbed sentences could serve as robustness probes, checking whether frames and trees stay stable under minor edits, rather than as labelled training data whose gold may no longer hold. - **I5** (tokens): A reversible normalisation layer for variant characters (quotes, dashes, typo-prone forms) in crate en could shrink sparsity while keeping the original surface form for explanation. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Emna Fsih, Sameh Kchaou, Rahma Boujelbane and Lamia Hadrich-Belguith. 2022. Benchmarking transfer learning approaches for sentiment analysis of Arabic dialect. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{fsih2022benchmarking, title = {{Benchmarking transfer learning approaches for sentiment analysis of Arabic dialect}}, author = {Emna Fsih, Sameh Kchaou, Rahma Boujelbane and Lamia Hadrich-Belguith}, booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)}, year = {2022}, doi = {10.18653/v1/2022.wanlp-1.44}, url = {https://doi.org/10.18653/v1/2022.wanlp-1.44} } ``` - [← previous ArNLI: Arabic Natural Language Inference for Entailment and Contradiction Detection](https://md.research.elamur.ai/free/en/papers/jallad-2022-arnli-arabic-natural-language-inference/) - [next → BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model](https://md.research.elamur.ai/free/en/papers/stephenson-2022-bert-he-predict-contrastive-focus/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/7/) ## References 13 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. [Arbert & marbert: deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). arXiv preprint arXiv:2101.01785. [↗](https://arxiv.org/abs/2101.01785) 2. Muhammad Abdul-Mageed, Chiyu Zhang, Abdelrahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. The nuanced arabic dialect identification shared task. In Proceedings of the Seventh Workshop for [Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/papers/habash-2022-arabic-natural-language-processing/) at EMNLP 2022, Abu Dhabi. 3. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. [Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic](https://md.research.elamur.ai/free/en/papers/farha-2021-overview-wanlp-2021-shared-task/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. 4. Abeer Abuzayed and Hend Al-Khalifa. 2021. [Sarcasm and sentiment detection in arabic tweets using bert-based models and data augmentation](https://md.research.elamur.ai/free/en/papers/abuzayed-2021-sarcasm-sentiment-detection-arabic-tweets/). In Proceedings of the sixth Arabic natural language processing workshop, pages 312–317. Methods in Natural Language Processing. Association for Computational Linguistics. 5. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics. 6. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T. Al-Natsheh. 2020. [Multi-dialect arabic bert for country-level dialect identification](https://md.research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/). 7. Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2019. Few-shot text classification with distributional signatures. arXiv preprint arXiv:1908.06039. [↗](https://arxiv.org/abs/1908.06039) 8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805) 9. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. 10. Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, and Wei Zhang. 2021. Don’t miss the labels: Label-semantic augmented meta-learner for few-shot text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2773– 2782, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiaoyang+Luo%2C+Lingqiao+Liu%2C+Yuhao+Lin%2C+and+Wei+Zhang%2E+2021%2E+Don%E2%80%99t+miss+the+labels%3A+Label%2Dsemantic+augmented+meta%2Dlearner+for+few%2Dshot+text+classification%2E+In+Findings+of+the+Association+for+Computatio) 11. Leila Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An Algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1202–1210, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Leila+Moudjari%2C+Karima+Akli%2DAstouati%2C+and+Farah+Benamara%2E+2020%2E+An+Algerian+corpus+and+an+annotation+platform+for+opinion+and+emotion+analysis%2E+In+Proceedings+of+the+12th+Language+Resources+and+Evalua) 12. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. [CAMeL tools: An open source python toolkit for Arabic natural language processing](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/). In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. 13. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical *Updated 2 November 2026*