--- title: "TAO at StanceEval2024 Shared Task — Melhem 2024" description: "The paper describes TAO, a team entry for the StanceEval2024 shared task at ArabicNLP 2024." authors: ["Anas Melhem", "Osama Hamed", "Thaer Sammar"] year: 2024 venue: "Proceedings of the Second Arabic Natural Language Processing Conference" source: "https://aclanthology.org/2024.arabicnlp-1.100/" doi: "10.18653/v1/2024.arabicnlp-1.100" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/melhem-2024-tao-stanceeval2024-shared-task-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/melhem-2024-tao-stanceeval2024-shared-task-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT # TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT **Authors:** Anas Melhem, [Osama Hamed](https://md.research.elamur.ai/free/en/authors/osama-hamed/), Thaer Sammar · **Year:** 2024 · **Venue:** Proceedings of the Second Arabic Natural Language Processing Conference · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.100/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.100.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.100](https://doi.org/10.18653/v1/2024.arabicnlp-1.100) · [OpenAlex](https://openalex.org/W4402683895) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/melhem-2024-tao-stanceeval2024-shared-task-arabic/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper describes TAO, a team entry for the StanceEval2024 shared task at ArabicNLP 2024. The task is to detect a tweet's stance toward a target (Favor, Against, None) in noisy, multi-dialectal Arabic social media text. The data is the Mawqif corpus: just over 4,000 tweets on three targets ("COVID-19 vaccine", "digital transformation", "women empowerment"). Each tweet is also annotated for sentiment and sarcasm, but TAO ignores those labels and uses a single-task setup. After cleaning the tweets, the authors fine-tuned AraBERT v1 with a softmax classifier on the [CLS] vector. They chose v1 over v2 because it scored better in their own trials. On their own 10% held-out split, TAO scored above 90 macro-F1 on every target. On the organizers' blind test set it dropped to between 64.55 and 73.30. The paper is inconsistent about which of the two lower scores belongs to COVID-19 versus digital transformation. The work is a simple, reproducible baseline (the code is public). Its main lesson is how far an in-house evaluation can overestimate real performance. ## Key points - **Data.** Mawqif has 4,121 tweets: 3,502 in train and 619 in test. Class balance differs by target: - COVID-19 vaccine is roughly balanced between Favor and Against (about 43.6% each). - Digital transformation (about 77% Favor) and women empowerment (about 64% Favor) lean heavily toward Favor. - The None class is rare: about 5–13% depending on target. - **Preprocessing.** The authors removed: - numbers, Latin letters and newlines; - any token containing "_", "#" or "@"; - diacritics, Tatweel and other non-Arabic characters. - **Model and training.** AraBERT v1 (BERT-Base), using the BertTokenizer and Hugging Face TrainingArguments. Settings: batch size 16 for training and 32 for evaluation, 3 epochs, max length 128, learning rate in the 2e-5 to 5e-5 range. Training ran on Google Colab. - **Internal results.** On a 70/20/10 train/validation/test split, macro-F1 was: - COVID-19 vaccine: 90.36; - the digital transformation target: 92.36; - the women empowerment target: 91.34. These beat CAMeLBERT-da, MARBERT, AraBERT and AraBERT-twitter as reported by Alturayeif et al. (2022). AraBERT v2 reached 73.05, 60.14 and 82.02 on the same targets. - **Blind-test results.** Scores were 73.30 for women empowerment, with 70.51 and 64.55 for the other two targets (the assignment is swapped between abstract and results). The official macro-F1 averages only the Favor and Against F1 scores. ## Interesting ideas - The official metric averages F1 over the two opinion-bearing classes and leaves out the rare None class, so scoring focuses on the decisions that matter rather than on the neutral class. — For Mova (answer): Test scoring could report accuracy on content-bearing answers separately from "no answer / unknown" cases, so abstentions neither inflate nor hide quality. - A score above 90 on a self-made split fell to 64–73 on the organizers' blind set, which shows how misleading a homemade split can be. — For Mova (learn): Always keep a held-out set built by a different process (other source, other annotator) as a control, and treat large gaps between internal and external scores as a warning of leakage or overfitting. - The authors checked that per-target class proportions in train and test closely match before training, a cheap sanity check on the split. — For Mova (learn): When building silver data or splits from treebanks, automatically compare label distributions (deprels, frame types) across splits and flag skewed ones. - Social-media noise was handled by aggressively stripping hashtags, mentions, diacritics and foreign script before tokenization. This trades information for cleaner input. — For Mova (tokens): A configurable normalization layer for noisy English input (handles, hashtags, emoji) could sit before the tokenizer, but it should record what was removed so later stages stay explainable. - The corpus labels stance, sentiment and sarcasm on the same tweets, which opens the way to studying how these attitudes interact (the paper defers multi-task use to future work). — For Mova (frames): Speech-act and irony modules in prag could be evaluated jointly with stance-like attitude labels, checking whether detected irony correctly flips the inferred attitude of the speaker. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{melhem2024tao, title = {{TAO at StanceEval2024 Shared Task: Arabic Stance Detection using AraBERT}}, author = {Melhem, Anas and Hamed, Osama and Sammar, Thaer}, booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference}, year = {2024}, doi = {10.18653/v1/2024.arabicnlp-1.100}, url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.100} } ``` - [← previous Ta'keed: The First Generative Fact-Checking System for Arabic Claims](https://md.research.elamur.ai/free/en/papers/althabiti-2024-ta-keed-first-generative-fact/) - [next → Training LLMs to Recognize Hedges in Dialogues about Roadrunner Cartoons](https://md.research.elamur.ai/free/en/papers/paige-2024-training-llms-recognize-hedges-dialogues/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training BERT on arabic tweets: Practical considerations. CoRR, abs/2102.10684. [↗](https://arxiv.org/abs/2102.10684) [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/) 2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/) 3. Tahani Alqurashi. 2022. Stance analysis of distance education in the kingdom of saudi arabia during the COVID-19 pandemic using arabic twitter data. Sensors, 22(3):1006. [find](https://scholar.google.com/scholar?q=Tahani+Alqurashi%2E+2022%2E+Stance+analysis+of+distance+education+in+the+kingdom+of+saudi+arabia+during+the+COVID%2D19+pandemic+using+arabic+twitter+data%2E+Sensors%2C+22%283%29%3A1006%2E) 4. Reema Khaled AlRowais and Duaa AlSaeed. 2023. Arabic stance detection of covid-19 vaccination using transformer-based approaches: a comparison study. Arab Gulf Journal of Scientific Research. [find](https://scholar.google.com/scholar?q=Reema+Khaled+AlRowais+and+Duaa+AlSaeed%2E+2023%2E+Arabic+stance+detection+of+covid%2D19+vaccination+using+transformer%2Dbased+approaches%3A+a+comparison+study%2E+Arab+Gulf+Journal+of+Scientific+Research%2E) 5. Nora Saleh Alturayeif, Hamzah Luqman, and Moataz A. Ahmed. 2023. Enhancing stance detection through sequential weighted multi-task learning. Soc. Netw. Anal. Min., 14(1):7. [find](https://scholar.google.com/scholar?q=Nora+Saleh+Alturayeif%2C+Hamzah+Luqman%2C+and+Moataz+A%2E+Ahmed%2E+2023%2E+Enhancing+stance+detection+through+sequential+weighted+multi%2Dtask+learning%2E+Soc%2E+Netw%2E+Anal%2E+Min%2E%2C+14%281%29%3A7%2E) 6. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label Arabic dataset for target-specific stance detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nora+Saleh+Alturayeif%2C+Hamzah+Abdullah+Luqman%2C+and+Moataz+Aly+Kamaleldin+Ahmed%2E+2022%2E+Mawqif%3A+A+multi%2Dlabel+Arabic+dataset+for+target%2Dspecific+stance+detection%2E+In+Proceedings+of+the+Seventh+Arabic+Na) 7. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiao+Cheng%2C+Juntao+Liu%2C+Xiaoye+Qu%2C+Jin+Zhao%2C+Jiaqing+Liang%2C+Zhefeng+Wang%2C+Baoxing+Huai%2C+Nicholas+Jing+Yuan%2C+and+Yanghua+Xiao%2E+2021%2E+HacRED%3A+A+large%2Dscale+relation+extraction+dataset+toward+hard+cases+) 8. Imane Guellil, Houda Saadane, Faiçal Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. J. King Saud Univ. Comput. Inf. Sci., 33(5):497–507. [find](https://scholar.google.com/scholar?q=Imane+Guellil%2C+Houda+Saadane%2C+Fai%C3%A7al+Azouaou%2C+Billel+Gueni%2C+and+Damien+Nouvel%2E+2021%2E+Arabic+natural+language+processing%3A+An+overview%2E+J%2E+King+Saud+Univ%2E+Comput%2E+Inf%2E+Sci%2E%2C+33%285%29%3A497%E2%80%93507%2E) [our card](https://md.research.elamur.ai/free/en/papers/guellil-2019-arabic-natural-language-processing-overview/) 9. Fatima Haouari and Tamer Elsayed. 2023. Detecting stance of authorities towards rumors in arabic tweets: A preliminary study. In Advances in Information Retrieval - 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part II, volume 13981 of Lecture Notes in Computer Science, pages 430–438. Springer. [find](https://scholar.google.com/scholar?q=Fatima+Haouari+and+Tamer+Elsayed%2E+2023%2E+Detecting+stance+of+authorities+towards+rumors+in+arabic+tweets%3A+A+preliminary+study%2E+In+Advances+in+Information+Retrieval+%2D+45th+European+Conference+on+Informa) [our card](https://md.research.elamur.ai/free/en/papers/haouari-2023-detecting-stance-authorities-towards-rumors/) 10. Abdullah Al Imran and Md Nur Amin. 2021. Deep bangla authorship attribution using transformer models. In Computational Data and Social Networks -10th International Conference, CSoNet 2021, Virtual Event, November 15-17, 2021, Proceedings, volume 13116 of Lecture Notes in Computer Science, pages 118–128. Springer. [find](https://scholar.google.com/scholar?q=Abdullah+Al+Imran+and+Md+Nur+Amin%2E+2021%2E+Deep+bangla+authorship+attribution+using+transformer+models%2E+In+Computational+Data+and+Social+Networks+%2D10th+International+Conference%2C+CSoNet+2021%2C+Virtual+Eve) 11. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2024. A survey on arabic named entity recognition: Past, recent advances, and future trends. IEEE Trans. Knowl. Data Eng., 36(3):943–959. [find](https://scholar.google.com/scholar?q=Xiaoye+Qu%2C+Yingjie+Gu%2C+Qingrong+Xia%2C+Zechang+Li%2C+Zhefeng+Wang%2C+and+Baoxing+Huai%2E+2024%2E+A+survey+on+arabic+named+entity+recognition%3A+Past%2C+recent+advances%2C+and+future+trends%2E+IEEE+Trans%2E+Knowl%2E+Data+En) [our card](https://md.research.elamur.ai/free/en/papers/qu-2023-survey-arabic-named-entity-recognition/) 12. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2020%2E+AraBERT%3A+Transformer%2Dbased+model+for+Arabic+language+understanding%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Tool) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/) 13. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2021%2E+AraELECTRA%3A+Pre%2Dtraining+text+discriminators+for+Arabic+language+understanding%2E+In+Proceedings+of+the+Sixth+Arabic+Natural+Language+Processing+Workshop%2C) 14. Gilbert Badaro, Ramy Baly, Hazem M. Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al Sallab, and Ali Hamdi. 2019. A survey [find](https://scholar.google.com/scholar?q=Gilbert+Badaro%2C+Ramy+Baly%2C+Hazem+M%2E+Hajj%2C+Wassim+El%2DHajj%2C+Khaled+Bashir+Shaban%2C+Nizar+Habash%2C+Ahmad+Al+Sallab%2C+and+Ali+Hamdi%2E+2019%2E+A+survey)