---
title: "Ensemble of pre-trained language models and — Daouadi 2024"
description: "The authors address fine-grained hate speech classification for Arabic tweets."
authors: ["Kheir Eddine Daouadi", "Yaakoub Boualleg", "Kheir Eddine Haouaouchi"]
year: 2024
venue: "arXiv:2407.02448"
source: "https://arxiv.org/abs/2407.02448"
doi: "10.48550/arxiv.2407.02448"
arxiv: "2407.02448"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/daouadi-2024-ensemble-pre-trained-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/daouadi-2024-ensemble-pre-trained-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Ensemble of pre-trained language models and data augmentation for hate speech detection from Arabic tweets

# Ensemble of pre-trained language models and data augmentation for hate speech detection from Arabic tweets

**Authors:** Kheir Eddine Daouadi, Yaakoub Boualleg, Kheir Eddine Haouaouchi · **Year:** 2024 · **Venue:** arXiv:2407.02448 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.02448) · [PDF](https://arxiv.org/pdf/2407.02448) · [DOI 10.48550/arxiv.2407.02448](https://doi.org/10.48550/arxiv.2407.02448) · [arXiv 2407.02448](https://arxiv.org/abs/2407.02448) · [OpenAlex](https://openalex.org/W4400374489)

**Citations:** 2 citations (openalex-snapshot · 2026-10-14) · 2 in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors address fine-grained hate speech classification for Arabic tweets. Two obstacles limit earlier systems: modest accuracy and a strong class imbalance, where the rare hate categories are badly underrepresented. The task uses a benchmark dataset with five labels: non-hate, general hate, racial, religious and sexism. The authors fine-tune three Arabic BERT variants pre-trained on Twitter text (bert-base-arabertv02-twitter, bert-large-arabertv02-twitter, MARBERT) and combine them by majority voting and by average voting. To ease the imbalance, they enlarge the training set in two ways. First, they directly add religious-hate tweets taken from other published corpora. Second, they apply semi-supervised pseudo-labelling: their trained model assigns the five-way labels to tweets from other previously annotated hate-speech datasets, and those tweets join the training data. Majority voting beats each single model, and augmentation improves the result further, with the clearest gains on the minority classes. The work suggests that cheap ensembling plus reuse of existing annotated corpora can partly compensate for scarce labels.

## Key points
- Hyperparameter search settled on a learning rate of 1e-5 for all three models. Epochs were 4, 2 and 3 and batch sizes were 16, 8 and 8 for base-twitter, large-twitter and MARBERT.
- Single models reached about 84.2–84.7 weighted F1. The ensembles scored 85.10 with average voting and 85.48 with majority voting, where macro F1 was 72.66.
- Religious-hate tweets from Albadi et al. (2019) and Alsafari et al. (2020c) were added as they were. Pseudo-labelling of other labelled sets produced 1034 general hate, 950 religious, 863 sexism and 926 racial tweets.
- With augmentation, weighted F1 rose to 85.65 and macro F1 to 73.80. Racial F1 went from 53.85 to 57.25 and religious from 82.56 to 83.50, while non-hate dipped slightly (92.73 to 92.41).
- Per-class results beat the earlier data augmentation methods the authors compared against, for example racial 57.25 versus 44.15–50.99.

## Interesting ideas
- Hard majority voting over several differently pre-trained models helped most on the weakest, rarest classes, beyond what soft probability averaging gave. Disagreement between diverse models seems to cancel class-specific errors. — For Mova (learn): when several witness parsers produce silver trees, a plain per-arc majority vote is a simple baseline to try. Its gains should be checked separately on rare relations, not only on overall LAS.
- Corpora annotated under a different or coarser label scheme can be relabelled by one's own model into the target scheme, which turns neighbouring resources into extra minority-class data. — For Mova (learn): treebanks or corpora with incompatible annotation could be re-parsed by gparse to yield extra silver examples of rare constructions.
- The paper separates two import routes. Data whose labels map cleanly onto the target classes (religious hate) is added directly, and everything else goes through model relabelling. — For Mova (tree): in UD dialect conversion, deterministically mappable labels could be converted by rule, and only ambiguous ones sent to a parser or LLM teacher for relabelling.
- Reporting per-class and macro F1 next to weighted F1 showed that augmentation mainly helps small classes and slightly hurts the dominant one, which a single aggregate number would have hidden. — For Mova (learn): evaluations of new silver data should report per-relation or per-frame scores, so that gains on rare cases and losses on frequent ones both stay visible.
- The models were chosen because they were pre-trained on the same genre as the task (tweets). That choice targets the informal text the classifier actually sees. — For Mova (tree): if Mova must read informal or dialogue-heavy English, an encoder pre-trained on that genre could back gparse, and its effect could be measured against a general-domain encoder.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{daouadi2024ensemble,
  title = {{Ensemble of pre-trained language models and data augmentation for hate speech detection from Arabic tweets}},
  author = {Daouadi, Kheir Eddine and Boualleg, Yaakoub and Haouaouchi, Kheir Eddine},
  year = {2024},
  doi = {10.48550/arxiv.2407.02448},
  eprint = {2407.02448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.02448}
}
```

- [← previous Enhancing TTS Stability in Hebrew using Discrete Semantic Units](https://md.research.elamur.ai/free/en/papers/langstream-2024-zeldes-enhancing-tts-stability-hebrew-using/)
- [next → Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

9 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Al-Hassan, H. Al-Dossari, Detection of hate speech in Arabic tweets using deep learning. Multimedia Systems (2021). https://doi.org/10.1007/s00530-020-00742-w. R. Duwairi, A. Hayajneh, M. Quwaider, A Deep Learning Framework for Automatic Detection of Hate Speech Embedded in Arabic Tweets. Arabian Journal for Science and Engineering, 46(4), 4001-4014, (2021). H. Mubarak, A. Rashed, K. Darwish, Y. Samih, A. Abdelali, Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192, (2021). K. E. Daouadi, R. Z. Rebaï, I. Amous, Organization vs. Individual: Twitter User classification. In International Workshop on Language Processing and Knowledge Management, (2018). [↗](https://doi.org/10.1007/s00530-020-00742-w) [our card](https://md.research.elamur.ai/free/en/papers/mubarak-2021-arabic-offensive-language-twitter-analysis/)
2. Elmadany, A., Zhang, C., Abdul-Mageed, M., & Hashemi, A. (2020). Leveraging affective bidirectional transformers for offensive language detection. arXiv preprint arXiv:2006.01266. W. Antoun, F. Baly, H. Hajj, Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104, (2020). [↗](https://arxiv.org/abs/2006.01266) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
3. Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2020). ARBERT & MARBERT: deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785. [↗](https://arxiv.org/abs/2101.01785) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
4. Alshaalan, R., & Al-Khalifa, H. (2020, December). Hate speech detection in saudi twittersphere: A deep learning approach. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (pp. 12-23). [find](https://scholar.google.com/scholar?q=Alshaalan%2C+R%2E%2C+%26+Al%2DKhalifa%2C+H%2E+%282020%2C+December%29%2E+Hate+speech+detection+in+saudi+twittersphere%3A+A+deep+learning+approach%2E+In+Proceedings+of+the+Fifth+Arabic+Natural+Language+Processing+Workshop+%28pp%2E+1)
5. Keleg, A., El-Beltagy, S. R., & Khalil, M. (2020, May). ASU_OPTO at OSACT4-Offensive Language Detection for Arabic text. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 66-70). [find](https://scholar.google.com/scholar?q=Keleg%2C+A%2E%2C+El%2DBeltagy%2C+S%2E+R%2E%2C+%26+Khalil%2C+M%2E+%282020%2C+May%29%2E+ASU%5FOPTO+at+OSACT4%2DOffensive+Language+Detection+for+Arabic+text%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing)
6. Abdellatif, M., & Elgammal, A. (2020, May). Offensive language detection in Arabic using ULMFiT. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 82-85). K. E. Daouadi, R. Z. Rebaï, I. Amous, Towards a Statistical Approach for User Classification in Twitter. In International Conference on Machine Learning for Networking, (2018) 33-43. [find](https://scholar.google.com/scholar?q=Abdellatif%2C+M%2E%2C+%26+Elgammal%2C+A%2E+%282020%2C+May%29%2E+Offensive+language+detection+in+Arabic+using+ULMFiT%2E+In+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Tools%2C+with+a+Shared+Ta)
7. Aldjanabi, W., Dahou, A., Al-qaness, M. A., Elaziz, M. A., Helmi, A. M., & Damaševičius, R. (2021, October). Arabic Offensive and Hate Speech Detection Using a Cross-Corpora Multi-Task Learning Model. In Informatics (Vol. 8, No. 4, p. 69). MDPI. [find](https://scholar.google.com/scholar?q=Aldjanabi%2C+W%2E%2C+Dahou%2C+A%2E%2C+Al%2Dqaness%2C+M%2E+A%2E%2C+Elaziz%2C+M%2E+A%2E%2C+Helmi%2C+A%2E+M%2E%2C+%26+Dama%C5%A1evic%CC%8Cius%2C+R%2E+%282021%2C+October%29%2E+Arabic+Offensive+and+Hate+Speech+Detection+Using+a+Cross%2DCorpora+Multi%2DTask+Learning+Model)
8. Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., & Chowdhury, S. A. (2020, May). ALT submission for OSACT shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 61-65). [find](https://scholar.google.com/scholar?q=Hassan%2C+S%2E%2C+Samih%2C+Y%2E%2C+Mubarak%2C+H%2E%2C+Abdelali%2C+A%2E%2C+Rashed%2C+A%2E%2C+%26+Chowdhury%2C+S%2E+A%2E+%282020%2C+May%29%2E+ALT+submission+for+OSACT+shared+task+on+offensive+language+detection%2E+In+Proceedings+of+the+4th+Workshop+o)
9. Mubarak, H., Darwish, K., & Magdy, W. (2017, August). Abusive language detection on Arabic social media. In Proceedings of the first workshop on abusive language online (pp. 52-56). F. Husain, OSACT4 Shared Task on Offensive Language Detection: Intensive Preprocessing-Based Approach. arXiv preprint arXiv:2005.07297, (2020). [↗](https://arxiv.org/abs/2005.07297) [our card](https://md.research.elamur.ai/free/en/papers/mubarak-2017-abusive-language-detection-arabic-social/)