---
title: "Overview of OSACT5 Shared Task on Arabic Offensive"
description: "The paper reports on the OSACT5 shared task, which follows up OSACT4 and asks… 0.852 F1. 12 references, cited by 15 papers here."
authors: ["Hamdy Mubarak", "Hend Al-Khalifa", "Abdulmohsen Al-Thubaity"]
year: 2022
venue: "Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection"
source: "https://aclanthology.org/2022.osact-1.20/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mubarak-2022-overview-osact5-shared-task-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mubarak-2022-overview-osact5-shared-task-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Overview of OSACT5 Shared Task on Arabic Offensive Language and Hate Speech Detection

# Overview of OSACT5 Shared Task on Arabic Offensive Language and Hate Speech Detection

**Authors:** [Hamdy Mubarak](https://md.research.elamur.ai/free/en/authors/hamdy-mubarak/), [Hend Al-Khalifa](https://md.research.elamur.ai/free/en/authors/hend-al-khalifa/) and [Abdulmohsen Al-Thubaity](https://md.research.elamur.ai/free/en/authors/abdulmohsen-al-thubaity/) · **Year:** 2022 · **Venue:** [Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks 2022](https://md.research.elamur.ai/free/en/venues/osact/2022/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.osact-1.20/) · [PDF](https://aclanthology.org/2022.osact-1.20.pdf)

**Citations:** [15](https://md.research.elamur.ai/free/en/papers/mubarak-2022-overview-osact5-shared-task-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper reports on the OSACT5 shared task, which follows up OSACT4 and asks systems to find abusive content in Arabic tweets at three levels of detail. Subtask A is a binary decision on whether a tweet is offensive. Subtask B decides whether it is hate speech. Subtask C assigns a hate speech type: race, religion, ideology, disability, social class, gender, or none.

The organisers reused a corpus of 12,698 tweets. These were collected by searching for emojis that often appear in abusive posts, then labelled by crowd workers on Appen. Systems were ranked by macro-averaged F1 on CodaLab.

Most participating teams fine-tuned Arabic or multilingual transformers, and the leading systems combined several of them in ensembles. Top scores were 0.852 F1 for offensiveness and 0.831 for hate speech. Fine-grained typing reached only 0.528, which shows that telling hate categories apart is still hard. The task gives a shared benchmark and a snapshot of current methods for Arabic content moderation.

## Key points
- Corpus: 12,698 tweets drawn from 4.4M Arabic tweets posted June 2016 – November 2017. All contained emojis from a list mined from earlier offensive-tweet datasets. Splits were 70% train, 10% dev, 20% test.
- Annotation: crowdsourced on Appen, where every tweet got 3 judgements and annotators had to score at least 80% on a set of 200 test questions. Cohen's kappa was 0.82. Labelling ran as a binary OFF/CLN job, followed by a second job that typed the offensive tweets.
- Participation: 40/17 teams signed up/submitted for A, 26/12 for B and 23/10 for C. Ten system papers were submitted and 8 were accepted.
- Best results: GOF reached F1 0.852 on A. iCompass reached F1 0.831 on B and 0.528 on C, with accuracy 0.919 on C despite the low macro F1.
- Methods: preprocessing included character normalisation and removing diacritics, repeated letters and non-Arabic tokens. Models included AraBERT, MARBERT, ARBERT, QARiB, AraElectra, mT5, XLMRoberta and others, with ensembles and a QRNN-based head among the winners.

## Interesting ideas
- Gathering candidate data through emojis linked to abuse, instead of keyword lists, captures the target phenomenon regardless of topic, dialect or genre.
- Annotation was split into a cheap binary pass, with fine-grained typing applied only to the positives, which saves effort on the many clean items.
- Macro F1 was the headline metric, backed by precision and recall on the positive class, so a model cannot win by predicting the majority label; Subtask C's high accuracy but low F1 illustrates this.
- Teams could submit many runs but had to name a primary and a secondary run, which allows honest comparison of setups without cherry-picking the best test score.
- Mentions, URLs and empty lines were replaced with fixed placeholders such as @USER and URL, and rare, closely related categories were merged so the label set stays learnable.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): similar non-lexical cues could seed a corpus of ironic or hostile speech acts for prag without biasing it toward specific trigger words.
- **I2** (learn): LLM teacher tiers could follow the same cascade, with a cheap filter deciding relevance and an expensive teacher labelling only the retained sentences.
- **I3** (answer): reader evaluation on school tests should report per-question-type macro scores so that rare question kinds are not hidden by common easy ones.
- **I4** (learn): fix the primary configuration before scoring on held-out data, and log the alternatives as documented ablations.
- **I5** (tokens): the en tokenizer could map such non-linguistic spans to typed placeholder tokens so that the tagger and parser see clean, consistent input.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (3): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Hamdy Mubarak, Hend Al-Khalifa and Abdulmohsen Al-Thubaity. 2022. Overview of OSACT5 Shared Task on Arabic Offensive Language and Hate Speech Detection. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mubarak2022overview,
  title = {{Overview of OSACT5 Shared Task on Arabic Offensive Language and Hate Speech Detection}},
  author = {Hamdy Mubarak, Hend Al-Khalifa and Abdulmohsen Al-Thubaity},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.20/}
}
```

- [← previous Improving Event Coreference Resolution Using Document-level and Topic-level Information](https://md.research.elamur.ai/free/en/papers/xu-2022-improving-event-coreference-resolution-document/)
- [next → The Universal Anaphora Scorer](https://md.research.elamur.ai/free/en/papers/yu-2022-universal-anaphora-scorer/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Table 1: Statistics and examples from the annotated corpus Class/Subclass Clean # 8,235 % 65 Example  ÓBAK. Qº ®K IÓXAÓ É ’ ¯@ Y « ùÊ« É’m' áË You won’t have a better tomorrow as long as you think about yesterday. éK QºË@ Q ®  JK èA‚« . È@ ñ‚ËAë úΫ ¼ñK. @ áªÊK (CLN or NOT_OFF) Offensive 4,463 (OFF) Hate Speech May God curse your father for this question! I hope this fool will die out! (Note: 30% of Offensive tweets are labeled as Hate Speech) 1,339  Ê ¯  ÐñJ Ë@ HA  JK. ÈAg. QË@ ‘ªK. QêªË ɓñ K AÓ é\<Ë@ð . H. X @ HCJ Girls today are impolite. I swear to God, we don’t reach for some men immorality. á J.Ê ®Ë@ð ú æ  K@ ¼ Y gAK é\<Ë@ . Ê jJÓ ÕºJ.ª ƒ  - Gender - Race People of your country are musty. May God take (kill) you and the Philippines. - Ideology You club is vile, no doubt about that. Your party cannot do anything except barking. - Social Class As you are a resident, shut up and let original citizens speak. Swallow, plumber! - Religion - Disability Team aiXplain (Alzu’bi et al., 2022) iCompass (Ben Nessir et al., 2022) [find](https://scholar.google.com/scholar?q=Table+1%3A+Statistics+and+examples+from+the+annotated+corpus+Class%2FSubclass+Clean+%23+8%2C235+%25+65+Example+%10+%C2%81%06%06%C3%93BAK%2E+Q%C2%BA%06%06+%C2%AE%10K+I%06%06%C3%93XA%C3%93+%C3%89+%C2%92%06%06+%C2%AF%40+Y%13%06%06+%C2%AB+%C3%B9%06%06%C3%8A%C2%AB+%C3%89%06%06%C2%92%06%06m%1A%10%27+%C3%A1%C3%8B+You+won%E2%80%99t+have+a+better+tomorrow+as)
2. AlexU-AIC (Shapiro et al., 2022) [find](https://scholar.google.com/scholar?q=AlexU%2DAIC+%28Shapiro+et+al%2E%2C+2022%29)
3. CHILLAX (Makram et al., 2022) [find](https://scholar.google.com/scholar?q=CHILLAX+%28Makram+et+al%2E%2C+2022%29)
4. GUCT (Elkaref and Abu-Elkheir, 2022) [find](https://scholar.google.com/scholar?q=GUCT+%28Elkaref+and+Abu%2DElkheir%2C+2022%29)
5. Meta-AI (AlKhamissi and Diab, 2022) UPV (de Paula et al., 2022) Table 2: List of participating teams in Subtasks A, B, and C (alphabetical order) [find](https://scholar.google.com/scholar?q=Meta%2DAI+%28AlKhamissi+and+Diab%2C+2022%29+UPV+%28de+Paula+et+al%2E%2C+2022%29+Table+2%3A+List+of+participating+teams+in+Subtasks+A%2C+B%2C+and+C+%28alphabetical+order%29)
6. Elkaref, N. and Abu-Elkheir, M. (2022). Guct at arabic hate arXiv:2004.02192. speech 2022: Towards a better isotropy for hate speech Mubarak, H., Hassan, S., and Chowdhury, S. A. (2022). detection. OSACT, 5. [Emojis as anchors to detect arabic offensive language and hate speech](https://md.research.elamur.ai/free/en/papers/mubarak-2022-emojis-anchors-detect-arabic-offensive/). arXiv preprint arXiv:2201.06723. [↗](https://arxiv.org/abs/2004.02192)
7. Makram, K. H., Nessim, K. G., Abd-Almalak, M. E., Shapiro, A., Khalafallah, A., and Torki, M. (2022). Alexu-Roshdy, S. Z., Salem, S. H., Thabet, F. F., and Mohamed, aic at arabic hate speech 2022: Contrast to classify. OS-E. [find](https://scholar.google.com/scholar?q=Makram%2C+K%2E+H%2E%2C+Nessim%2C+K%2E+G%2E%2C+Abd%2DAlmalak%2C+M%2E+E%2E%2C+Shapiro%2C+A%2E%2C+Khalafallah%2C+A%2E%2C+and+Torki%2C+M%2E+%282022%29%2E+Alexu%2DRoshdy%2C+S%2E+Z%2E%2C+Salem%2C+S%2E+H%2E%2C+Thabet%2C+F%2E+F%2E%2C+and+Mohamed%2C+aic+at+arabic+hate+speech+2022%3A+Con)
8. H. (2022). Chillax - at arabic hate speech 2022: A ACT, 5. hybrid machine learning and transformers based model to Watch, H. S. (2014). Hate speech watch. hate detect arabic offensive and hate speech. OSACT, 5. crimes: Consequences of hate speech. In Mostafa, A., Mohamed, O., and Ashraf, A. (2022). Gof [http://www.nohatespeechmovement.org/hate-at](http://www.nohatespeechmovement.org/hate-at) arabic hate speech 2022: Breaking the loss function speechwatch/focus/ consequences-of-hate-speech. convention for data-imbalanced arabic offensive text de-Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., tection. OSACT, 5. [↗](http://www.nohatespeechmovement.org/hate-at)
9. Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Mubarak, H., Darwish, K., and Magdy, W. (2017). [find](https://scholar.google.com/scholar?q=Karadzhov%2C+G%2E%2C+Mubarak%2C+H%2E%2C+Derczynski%2C+L%2E%2C+Pitenis%2C+Mubarak%2C+H%2E%2C+Darwish%2C+K%2E%2C+and+Magdy%2C+W%2E+%282017%29%2E)
10. Abusive Z., and Çöltekin, Ç. (2020). Semeval-2020 task 12: Mul-language detection on arabic social media. In Proceed-tilingual offensive language identification in social media ings of the first workshop on abusive language online, (offenseval 2020). arXiv preprint arXiv:2006.07235. pages 52–56. [↗](https://arxiv.org/abs/2006.07235)
11. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., and Al-Khalifa, H. (2020a). [Overview of osact4 arabic offensive language detection shared task](https://md.research.elamur.ai/free/en/papers/mubarak-2020-overview-osact4-arabic-offensive-language/). In Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection, pages 48–52.
12. Mubarak, H., Rashed, A., Darwish, K., Samih, Y., and Abdelali, A. (2020b). [Arabic offensive language on twitter: Analysis and experiments](https://md.research.elamur.ai/free/en/papers/mubarak-2021-arabic-offensive-language-twitter-analysis/). arXiv preprint 164 hAJ. JË@ úΫ B@ PY ®K  B ½K. Qk .½Ë X ú ¯ ½ ƒ  B ©J “ð ½K XA K  ½Ó@X  K@ Õæ ®Ó ! ¼AJ.ƒ AK ©ÊK. @ . àñÒʾJK á J ʓB@ P B@ Éë@ É gð Õ æº XñîE Bð èQ ®» A Jk@ ñë éJ Ë èY» A KñK. @ ½K X ¬ @ñÊÒªJK. @ñJ K@ Why are you doing this to us? Are we disbelievers or Jews?  @ X  Q.ªK ¬Q«AÓ . éË áK QK Ag. IªÊ£ Ð Q ®Ë@ This dwarf got two prizes, but he does not know how to express. Affiliation aiXplain Inc, USA iCompass, Tunisia Alexandria University, Egypt Helwan University, Egypt Helwan University, Egypt German University in Cairo, Egypt Meta, USA Universitat Politecnica de Valencia, Spain Subtasks A A, B, C A, B, C A, B A A A, B, C A, B, C Team aiXplain (Alzu’bi et al., 2022)

*Updated 1 November 2026*