---
title: "SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in"
description: "The paper tackles the ArAIEval 2024 unimodal propaganda task: given an Arabic news paragraph or… 12 references, cited by 1 paper here."
authors: ["Mary Fouad", "Julie Weeds"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.55/"
doi: "10.18653/v1/2024.arabicnlp-1.55"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fouad-2024-sussexai-araieval-shared-task-mitigating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fouad-2024-sussexai-araieval-shared-task-mitigating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/) / SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in Arabic Propaganda Detection

# SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in Arabic Propaganda Detection

**Authors:** Mary Fouad and [Julie Weeds](https://md.research.elamur.ai/free/en/authors/julie-weeds/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.55/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.55.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.55](https://doi.org/10.18653/v1/2024.arabicnlp-1.55) · [OpenAlex](https://openalex.org/W4402671508)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/fouad-2024-sussexai-araieval-shared-task-mitigating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles the ArAIEval 2024 unimodal propaganda task: given an Arabic news paragraph or tweet, systems must name the propaganda techniques used and locate their exact spans. The authors treat it as token-level sequence tagging. They map annotated spans back into their source sentences, handling overlapping spans one at a time, and fine-tune AraBERT for token classification.

The labels are heavily skewed. Two techniques dominate the spans, and most tokens carry no propaganda label at all. The authors try two fixes:
- **Synthetic minority examples:** one word of a span is masked, and the AraBERT MLM fills it in.
- **Random truncation:** the non-propaganda context around each span is cut down to a few tokens.

Augmentation lowered scores. Truncation doubled macro F1 (0.05 → 0.10) and raised micro F1 to 0.12, which placed fifth. The broader lesson is that micro F1 can hide whether rare classes are learned at all, so macro F1 should be reported next to it.

## Key points
- **Data:** ArAIEval24 comes as JSON, with each span given as start/end characters plus a technique label. Loaded language covers 55.72% of spans and name-calling 14.23%. Rare classes such as Appeal-to-Popularity, Red-Herring, Whataboutism and Straw-Man are each around 0.23%.
- **Model setup:** AutoModelForTokenClassification with the Hugging Face Trainer, learning rate 2e-5, 10 epochs, weight decay 0.01.
- **Augmentation:** each class received 1000 extra synthetic instances, so every class had between 1000 and 10,000 training instances.
- **Truncation:** only 0–5 random context tokens were kept on each side of a span. Non-propaganda tokens fell from 586,242 (about 89.05% of all tokens) to 120,539 (about 64.88%).
- **Test results (macro / micro F1):** AraBERT Basic 0.05 / 0.06, Augmented 0.03 / 0.05, Rand-Truncated 0.10 / 0.12. The baseline failed to predict more than 12 minority classes.

## Interesting ideas
- Shrinking the background context around labelled spans is a cheap way to undersample the dominant "no label" tag without discarding any positive examples.
- Masking each span word in turn and refilling it with an MLM backfired here, likely because it also multiplied background tokens and worsened the imbalance. Any augmentation should be checked for how it shifts the overall label mix.
- Report macro F1 and a per-class F1 chart next to micro F1, since a model that predicts only frequent classes can look competitive on micro F1 alone.
- Overlapping spans are handled by aligning each span to its own copy of the context sentence, avoiding a multi-label token scheme.
- The authors plan to use a detector that catches some rare classes to mine unannotated corpora, have humans curate the hits, and retrain.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When training span-style taggers on silver data (e.g., implicature or irony cues), trim unlabelled context around positives so the empty label does not swamp the rare ones.
- **I2** (learn): Before adding LLM- or MLM-generated training sentences, compare label distributions before and after, as a negative control on whether augmentation actually helps rare classes.
- **I3** (answer): Evaluate the readers per question type or skill, not just with one pooled accuracy, so rare reasoning types that silently fail stay visible.
- **I4** (frames): Overlapping frame or speech-act spans could be projected one at a time onto separate copies of the sentence, keeping each annotation simple and traceable.
- **I5** (learn): A weak detector for rare constructions could harvest candidate sentences from large text for teacher-tier or human vetting, bootstrapping coverage of rare constructions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Sussex](https://md.research.elamur.ai/free/en/universities/university-of-sussex/) — University, Brighton, United Kingdom · 22 works · 3,332 citations
- [Minia University](https://md.research.elamur.ai/free/en/universities/minia-university/) — University, Minya, Egypt · 3 works · 134 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Mary Fouad and Julie Weeds. 2024. SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in Arabic Propaganda Detection. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{fouad2024sussexai,
  title = {{SussexAI at ArAIEval Shared Task: Mitigating Class Imbalance in Arabic Propaganda Detection}},
  author = {Mary Fouad and Julie Weeds},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.55},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.55}
}
```

- [← previous Survey on Computational Approaches to Implicature](https://md.research.elamur.ai/free/en/papers/anuranjana-2024-survey-computational-approaches-implicature/)
- [next → The church and the manor: Assessing and comparing the effects of language contact on two Middle English lexical domains](https://md.research.elamur.ai/free/en/papers/mambelli-2024-church-manor-assessing-comparing-effects/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/13/)

## References

12 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Firoj Alam, Abul Hasnat, Fatema Ahmed, Md Arid Hasan, and Maram Hasanain. 2024. [ArMeme: Propagandistic content in arabic memes](https://md.research.elamur.ai/free/en/papers/alam-2024-armeme-propagandistic-content-arabic-memes/). arXiv preprint arXiv:2406.03916. [↗](https://arxiv.org/abs/2406.03916)
2. Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Sajjad, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, Abdulaziz Al-Homaid, Wajdi Zaghouani, Tommaso Caselli, Gijs Danoe, Friso Stolk, Britt Bruntink, and Preslav Nakov. 2021. Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 611–649, Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2005.00033)
3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. [↗](https://doi.org/10.48550/arxiv.2003.00104)
4. Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Inf. Process. Manag., 56(5):1849–1864. Maram Hasanain, Md. Arid Hasan, Fatema Ahmed, Reem Suwaileh, Md. [↗](https://doi.org/10.1016/j.ipm.2019.03.005)
5. Rafiul Biswas, Wajdi Zaghouani, and Firoj Alam. 2024b. [ArAIEval Shared Task: Propagandistic techniques detection in unimodal and multimodal arabic content](https://md.research.elamur.ai/free/en/papers/hasanain-2024-araieval-shared-task-propagandistic-techniques/). In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok. Association for Computational Linguistics.
6. OpenAI. 2023. [GPT-4: Technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Technical report, OpenAI. [↗](https://doi.org/10.4230/lipics.cosit.2024.11)
7. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. [Truth of varying shades: Analyzing language in fake news and political fact-checking](https://md.research.elamur.ai/free/en/papers/rashkin-2017-truth-varying-shades-analyzing-language/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937, Copenhagen, Denmark. Association for Computational Linguistics.
8. Prashanth Vijayaraghavan and Soroush Vosoughi. 2022. TWEETSPIN: Fine-grained propaganda detection in social media using multi-view representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3433–3448, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48448/kxxk-zp83)
9. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. [SemEval-2020 task 11: Detection of propaganda techniques in news articles](https://md.research.elamur.ai/free/en/papers/martino-2020-semeval-2020-task-11-detection/). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). International Committee for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2009.02696)
10. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). [find](https://scholar.google.com/scholar?q=Giovanni+Da+San+Martino%2C+Seunghak+Yu%2C+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2C+Rostislav+Petrov%2C+and+Preslav+Nakov%2E+2019%2E+Fine%2Dgrained+analysis+of+propaganda+in+news+article%2E+In+Proceedings+of+the+2019+Conference+on+Em)
11. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. [SemEval-2021 task 6: Detection of persuasion techniques in texts and images](https://md.research.elamur.ai/free/en/papers/dimitrov-2021-semeval-2021-task-6-detection/). In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.semeval-1.7)
12. Maram Hasanain, Fatema Ahmed, and Firoj Alam. 2024a. Can GPT-4 identify propaganda? annotation and detection of propaganda spans in news articles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy. [find](https://scholar.google.com/scholar?q=Maram+Hasanain%2C+Fatema+Ahmed%2C+and+Firoj+Alam%2E+2024a%2E+Can+GPT%2D4+identify+propaganda%3F+annotation+and+detection+of+propaganda+spans+in+news+articles%2E+In+Proceedings+of+the+2024+Joint+International+Confer)

*Updated 1 November 2026*