---
title: "Team_Zero at StanceEval2024 — Galal 2024"
description: "The paper addresses stance detection on Arabic tweets, which means deciding whether a tweet supports, opposes, or is neutral toward a target."
authors: ["Omar Galal", "Abdelrahman Kaseb"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.89/"
doi: "10.18653/v1/2024.arabicnlp-1.89"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/galal-2024-team-zero-stanceeval2024-frozen-plms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/galal-2024-team-zero-stanceeval2024-frozen-plms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Team\_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection

# Team\_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection

**Authors:** Omar Galal, Abdelrahman Kaseb · **Institutions:** [Cairo University](https://md.research.elamur.ai/free/en/universities/cairo-university/) · **Year:** 2024 · **Venue:** Proceedings of the Second Arabic Natural Language Processing Conference · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.89/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.89.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.89](https://doi.org/10.18653/v1/2024.arabicnlp-1.89) · [OpenAlex](https://openalex.org/W4402671410)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/galal-2024-team-zero-stanceeval2024-frozen-plms/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper addresses stance detection on Arabic tweets, which means deciding whether a tweet supports, opposes, or is neutral toward a target. The work was a StanceEval2024 shared task entry. The authors avoid full fine-tuning and instead treat pre-trained language models as frozen feature extractors with a cheap classifier on top.

They used the Mawqif dataset, which has three targets: women empowerment, COVID vaccine, and digital transformation. Three families of features were compared:
- nine multilingual sentence transformers;
- five Arabic BERT variants, each pooled two ways ([CLS] vector or mean over all output tokens), with every embedding fed to scikit-learn logistic regression;
- a multi-task Parallel-Sum (P-SUM) head on a frozen MARBERT, trained jointly on stance and sentiment.

Arabic, tweet-pretrained encoders with mean pooling clearly outperformed the multilingual sentence models. The final system was a majority vote of the three strongest configurations. It scored 78.62 F avg2 on the blind test set and placed fifth of 16 teams. This suggests competitive results are possible without fine-tuning's compute and time cost.

## Key points
- **Data:** Mawqif has 3502 training tweets and 619 test tweets, labelled FAVOR / AGAINST / NONE, plus sentiment and sarcasm annotations.
  - Overall training distribution is roughly 61% favor, 29% against, 10% none.
  - Digital transformation is especially skewed, with about 77% favor.
- **Metric (F avg2):** for each target, average the F1 of FAVOR and AGAINST only, then average across the three targets.
- **Internal test (500 held-out examples):**
  - The best sentence transformer was paraphrase-multilingual-mpnet-base-v2 at 70.6; all-mpnet-base-v2 was lowest at 51.8.
  - MARBERT with averaged embeddings reached 76.7, P-SUM-MTL 76.3, and AraBERTv0.2-Twitter AVG 76.1.
- **Pooling:** mean pooling beat [CLS] for all five Arabic BERTs tested (AraBERTv0.2-Twitter, MARBERT, Qarib, ARBERT, Arabic-BERT). For example, MARBERT went from 71.5 with [CLS] to 76.7 with averaging.
- **Official test:** the ensemble scored 85.99 on women empowerment, 73.08 on COVID vaccine, 76.8 on digital transformation, and 78.62 overall (5th of 16).
- **Overfitting:** this was the main difficulty, caused by the small data split across three targets. One model on all targets beat per-target models for the sentence-transformer setup.

## Interesting ideas
- Averaging every token output of a frozen encoder gave a consistently better sentence vector than the [CLS] token, at no extra cost; this is a cheap default worth checking wherever frozen embeddings are pooled. — For Mova (tree): when gparse or auxiliary classifiers pool frozen transformer states for span- or sentence-level decisions, compare mean pooling against first-token pooling before adding parameters.
- A frozen encoder plus logistic regression is a fast, honest baseline that reveals how much a heavier trained head actually adds (here P-SUM barely moved past mean pooling). — For Mova (learn): keep a frozen-features + linear-probe baseline in evaluation so any gain from a new trained component is measured against what the raw representation already gives.
- Genre match of the pretraining corpus mattered more than whether the model was trained specifically for sentence embeddings, since tweet-pretrained Arabic BERTs beat purpose-built multilingual sentence models. — For Mova (tokens): when choosing or training encoders for tagging and lemmatization of fables, school texts, or word problems, prefer models whose pretraining genre resembles those texts over generic "sentence" models.
- Pooling training data across targets worked better than per-target models, because the way people signal support or opposition carries over between topics. — For Mova (frames): train speech-act, stance, or irony detectors in prag jointly across topics rather than per domain, treating the linguistic cues as shared and topic-independent.
- A simple majority vote over the three best, architecturally different systems produced the final submission. — For Mova (learn): the same vote over diverse witness parsers can decide silver labels, keeping only arcs or frames where most witnesses agree.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{galal2024team,
  title = {{Team\_Zero at StanceEval2024: Frozen PLMs for Arabic Stance Detection}},
  author = {Galal, Omar and Kaseb, Abdelrahman},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.89},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.89}
}
```

- [← previous Quranic Audio Dataset: Crowdsourced and Labeled Recitation from Non-Arabic Speakers](https://md.research.elamur.ai/free/en/papers/salameh-2024-quranic-audio-dataset-crowdsourced-labeled/)
- [next → Unifying the Scope of Bridging Anaphora Types in English: Bridging Annotations in ARRAU and GUM](https://md.research.elamur.ai/free/en/papers/levine-2024-unifying-scope-bridging-anaphora-types/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali%2C+Sabit+Hassan%2C+Hamdy+Mubarak%2C+Kareem+Darwish%2C+and+Younes+Samih%2E+2021%2E+Pre%2Dtraining+bert+on+arabic+tweets%3A+Practical+considerations%2E) [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/)
2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
3. Nora Alturayeif, Hamzah Luqman, Zaid Alyafeai, and Asma Yamani. 2024. Stanceeval 2024: The first arabic stance detection shared task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). [find](https://scholar.google.com/scholar?q=Nora+Alturayeif%2C+Hamzah+Luqman%2C+Zaid+Alyafeai%2C+and+Asma+Yamani%2E+2024%2E+Stanceeval+2024%3A+The+first+arabic+stance+detection+shared+task%2E+In+Proceedings+of+The+Second+Arabic+Natural+Language+Processing+Co) [our card](https://md.research.elamur.ai/free/en/papers/alturayeif-2024-stanceeval-2024-first-arabic-stance/)
4. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184. [find](https://scholar.google.com/scholar?q=Nora+Saleh+Alturayeif%2C+Hamzah+Abdullah+Luqman%2C+and+Moataz+Aly+Kamaleldin+Ahmed%2E+2022%2E+Mawqif%3A+A+multi%2Dlabel+arabic+dataset+for+target%2Dspecific+stance+detection%2E+In+Proceedings+of+the+The+Seventh+Arabi)
5. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+Arabert%3A+Transformer%2Dbased+model+for+arabic+language+understanding%2E+In+LREC+2020+Workshop+Language+Resources+and+Evaluation+Conference+11%E2%80%9316+May+2020%2C+page+9%2E) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
6. David R Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20(2):215–232. [find](https://scholar.google.com/scholar?q=David+R+Cox%2E+1958%2E+The+regression+analysis+of+binary+sequences%2E+Journal+of+the+Royal+Statistical+Society+Series+B%3A+Statistical+Methodology%2C+20%282%29%3A215%E2%80%93232%2E)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
8. Hugo Touvron et al. 2023. Llama 2: Open foundation and fine-tuned chat models. [find](https://scholar.google.com/scholar?q=Hugo+Touvron+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E)
9. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR, abs/2007.01852. [↗](https://arxiv.org/abs/2007.01852) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/)
10. Omar Galal, Ahmed H Abdel-Gawad, and Mona Farouk. 2024a. Federated freeze bert for text classification. Journal of Big Data, 11(1):28. [find](https://scholar.google.com/scholar?q=Omar+Galal%2C+Ahmed+H+Abdel%2DGawad%2C+and+Mona+Farouk%2E+2024a%2E+Federated+freeze+bert+for+text+classification%2E+Journal+of+Big+Data%2C+11%281%29%3A28%2E)
11. Omar Galal, Ahmed H Abdel-Gawad, and Mona Farouk. 2024b. Rethinking of bert sentence embedding for text classification. [find](https://scholar.google.com/scholar?q=Omar+Galal%2C+Ahmed+H+Abdel%2DGawad%2C+and+Mona+Farouk%2E+2024b%2E+Rethinking+of+bert+sentence+embedding+for+text+classification%2E)
12. Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. [find](https://scholar.google.com/scholar?q=Zeyu+Han%2C+Chao+Gao%2C+Jinyang+Liu%2C+Jeff+Zhang%2C+and+Sai+Qian+Zhang%2E+2024%2E+Parameter%2Defficient+fine%2Dtuning+for+large+models%3A+A+comprehensive+survey%2E) [our card](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
13. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E+Associ)
14. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2020%2E+Making+monolingual+sentence+embeddings+multilingual+using+knowledge+distillation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/)
15. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ali+Safaya%2C+Moutasem+Abdullatif%2C+and+Deniz+Yuret%2E+2020%2E+KUISAIL+at+SemEval%2D2020+task+12%3A+BERT%2DCNN+for+offensive+speech+identification+in+social+media%2E+In+Proceedings+of+the+Fourteenth+Workshop+on+Sema) [our card](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/)
16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771. [↗](https://arxiv.org/abs/1910.03771) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
17. Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2020. Improving bert performance for aspect-based sentiment analysis. arXiv preprint arXiv:2010.11731. [↗](https://arxiv.org/abs/2010.11731)
18. Abdelrahman Kaseb and Mona Farouk. 2022. SAIDS: A novel approach for sentiment analysis informed of dialect and sarcasm. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 22–30, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abdelrahman+Kaseb+and+Mona+Farouk%2E+2022%2E+SAIDS%3A+A+novel+approach+for+sentiment+analysis+informed+of+dialect+and+sarcasm%2E+In+Proceedings+of+the+Seventh+Arabic+Natural+Language+Processing+Workshop+%28WANL)
19. Abdelrahman Kaseb and Mona Farouk. 2023. Active learning for arabic sentiment analysis. Alexandria Engineering Journal, 77:177–187. [find](https://scholar.google.com/scholar?q=Abdelrahman+Kaseb+and+Mona+Farouk%2E+2023%2E+Active+learning+for+arabic+sentiment+analysis%2E+Alexandria+Engineering+Journal%2C+77%3A177%E2%80%93187%2E)
20. Abdelrahman Kaseb, Omar Galal, and Dina Elreedy. 2022. Analysis on tweets towards covid-19 pandemic: An application of text-based depression detection. In 2022 4th Novel Intelligent and Leading Emerging Sciences Conference (NILES), pages 131– 136. [find](https://scholar.google.com/scholar?q=Abdelrahman+Kaseb%2C+Omar+Galal%2C+and+Dina+Elreedy%2E+2022%2E+Analysis+on+tweets+towards+covid%2D19+pandemic%3A+An+application+of+text%2Dbased+depression+detection%2E+In+2022+4th+Novel+Intelligent+and+Leading+Emergi)
21. Hamdy Mubarak, Sabit Hassan, Shammur Absar Chowdhury, and Firoj Alam. 2022. Arcovidvac: Analyzing arabic tweets about covid-19 vaccination. arXiv preprint arXiv:2201.06496. [↗](https://arxiv.org/abs/2201.06496) [our card](https://md.research.elamur.ai/free/en/papers/mubarak-2022-arcovidvac-analyzing-arabic-tweets-about/)
22. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)