--- title: "PICT at StanceEval2024 — Shukla 2024" description: "The paper tackles three-way stance classification (Favor, Against, None) of Arabic tweets for the StanceEval 2024 shared task." authors: ["Ishaan Shukla", "Ankit Vaidya", "Geetanjali Kale"] year: 2024 venue: "Proceedings of the Second Arabic Natural Language Processing Conference" source: "https://aclanthology.org/2024.arabicnlp-1.99/" doi: "10.18653/v1/2024.arabicnlp-1.99" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shukla-2024-pict-stanceeval2024-stance-detection-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shukla-2024-pict-stanceeval2024-stance-detection-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models # PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models **Authors:** Ishaan Shukla, Ankit Vaidya, Geetanjali Kale · **Year:** 2024 · **Venue:** Proceedings of the Second Arabic Natural Language Processing Conference · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.99/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.99.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.99](https://doi.org/10.18653/v1/2024.arabicnlp-1.99) · [OpenAlex](https://openalex.org/W4402671370) **Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [2](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper tackles three-way stance classification (Favor, Against, None) of Arabic tweets for the StanceEval 2024 shared task. The tweets cover three topics: COVID-19 Vaccine, Digital Transformation and Women Empowerment. The data is the MAWQIF dataset, where each tweet also carries sarcasm and sentiment labels. The authors fine-tuned Arabic-specific BERT-style encoders (MARBERT, MARBERTv2, AraBERT, CamelBERT, QARiB and variants) in two ways: - **Single-task:** stance only. - **Multi-task:** stance as the main objective, with sarcasm and sentiment as auxiliary heads trained under a weighted combined loss. Preprocessing removed links, mentions and English words, and replaced emojis with their Arabic descriptions. The authors then searched every combination of their strongest single-task and multi-task models and merged the chosen ones by majority vote. The final ensemble reached a macro F1 of 78.02% on the blind test set, placing 7th. This shows that combining small dialect-aware encoders with auxiliary tasks is a practical recipe for low-data Arabic opinion mining. The "None" class remained the hardest category. ## Key points - **Data:** MAWQIF has 3502 labelled tweets, split 80:20 into 2805 training and 697 validation examples, plus a 619-tweet test set. Labels are skewed: Favor has 2148 examples and None only 333. - **Multi-task loss:** stance carries four times the weight of each auxiliary task, using (sentiment + 4·stance + sarcasm) / 6. Multi-task models generally beat single-task ones on validation but sometimes overfit more. - **Experiments:** 13 backbone models were tested via HuggingFace. Weighted random sampling against class imbalance gave no gain, so it was dropped. - **Final ensemble:** multi-task MARBERT, single-task AraBERT and single-task CamelBERT, combined by hard voting. Ties go to the model with the best validation F1. - **Test results:** 78.02% overall macro F1. By topic, Women Empowerment scored 75.52%, COVID-19 Vaccine 79.85% and Digital Transformation 78.69%. The official metric averages only the Favor and Against classes. ## Interesting ideas - Train correlated side labels (sarcasm, sentiment) as extra heads, but weight the main objective several times higher so the side tasks shape the shared encoder without dominating it. — For Mova (tree): gparse could share its encoder with auxiliary heads such as UPOS or morphological features while keeping arc/label loss dominant, which may regularize the parser on small treebanks. - When an ensemble splits without a majority, the system falls back to the single member with the best validation score instead of choosing at random. — For Mova (learn): when witness parsers disagree on a silver arc, the most reliable witness could serve as the tiebreaker, and the token could be flagged as low-agreement rather than silently dropped. - Ensemble membership was picked by testing every subset of top candidates on validation data, and the winning mix included both single-task and multi-task models. — For Mova (learn): an exhaustive subset search over witness parsers could reveal which combination yields the cleanest silver data, measured against a held-out gold set. - Emojis were rewritten as textual descriptions so a word-based model can read their meaning instead of treating them as unknown symbols. — For Mova (tokens): the tokenizer could map emojis and similar symbols to lemma-like descriptions, giving the tagger and later stages an interpretable word to work with. - The neutral "None" class failed most often, which the authors attribute to its lack of distinctive wording, unlike clear support or opposition. — For Mova (frames): stance or speech-act modules should treat "no stance" as an absence of explicit evaluative frames rather than a lexical pattern, and evaluation should report it separately. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cited by Cited by 2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet. - [StanceEval 2024: The First Arabic Stance Detection Shared Task](https://md.research.elamur.ai/free/en/papers/alturayeif-2024-stanceeval-2024-first-arabic-stance/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{shukla2024pict, title = {{PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models}}, author = {Shukla, Ishaan and Vaidya, Ankit and Kale, Geetanjali}, booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference}, year = {2024}, doi = {10.18653/v1/2024.arabicnlp-1.99}, url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.99} } ``` - [← previous Picking Up Where the Linguist Left Off: Mapping Morphology to Phonology through Learning the Residuals](https://md.research.elamur.ai/free/en/papers/aclbulk-2024-khalifa-picking-up-where-linguist-left/) - [next → Podcasting and input and output-based language learning of English modal verbs](https://md.research.elamur.ai/free/en/papers/alipour-2024-podcasting-input-output-based-language/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. *Figure 4: Confusion matrix of 3-way classification for ’Covid Vaccine’* [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali%2C+Sabit+Hassan%2C+Hamdy+Mubarak%2C+Kareem+Darwish%2C+and+Younes+Samih%2E+2021%2E+Pre%2Dtraining+bert+on+arabic+tweets%3A+Practical+considerations%2E+%2AFigure+4%3A+Confusion+matrix+of+3%2Dway+classification+f) [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/)