--- title: "Offensive language detection in Arabic using ULMFiT (2020)" description: "The paper describes an entry for the OSACT4 shared task (OffenseEval 2020) on Arabic tweets. 16 references, cited by 3 papers here." authors: ["Mohamed Abdellatif", "Ahmed Elgammal"] year: 2020 venue: "Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection" source: "https://aclanthology.org/2020.osact-1.13/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/abdellatif-2020-offensive-language-detection-arabic-ulmfit/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abdellatif-2020-offensive-language-detection-arabic-ulmfit/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/9/) / Offensive language detection in Arabic using ULMFiT # Offensive language detection in Arabic using ULMFiT **Authors:** Mohamed Abdellatif and Ahmed Elgammal · **Year:** 2020 · **Venue:** [Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks 2020](https://md.research.elamur.ai/free/en/venues/osact/2020/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.osact-1.13/) · [PDF](https://aclanthology.org/2020.osact-1.13.pdf) · [OpenAlex](https://openalex.org/W3088009796) **Citations:** 4 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/abdellatif-2020-offensive-language-detection-arabic-ulmfit/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes an entry for the OSACT4 shared task (OffenseEval 2020) on Arabic tweets. The task has two binary labels: whether a tweet is offensive, and whether it is hate speech. Both label sets are heavily skewed toward the harmless class. The authors start from a ULMFiT language model that Khooli (2019) had already pre-trained on Arabic Wikipedia. They adapt it to the tweet text and then train classifiers on top using fastai with an AWD-LSTM backbone. Following the original ULMFiT recipe, they build separate left-to-right and right-to-left models and average their predictions. Preprocessing is deliberately generic: whitespace splitting, a frequency cutoff, and fastai special tokens. On the development data, combining the two directions gave the best macro F1 in both subtasks. On the test set, offensive detection reached a macro F1 of 77. Hate detection lagged at 58, largely because the system recalled few minority-class tweets. The work shows that ULMFiT-style transfer is a workable, low-effort baseline for Arabic. It also shows that accuracy and weighted scores hide poor performance on rare classes. ## Key points - **Data:** 10k Arabic tweets from OSACT4, split 7k train / 1k dev / 2k test, with each tweet labelled for both subtasks. - Subtask A (offensive): roughly 1.4k positive tweets in train, 179 in dev. - Subtask B (hateful): roughly 0.4k positive tweets in train, only 44 in dev. - **Model and setup:** - Backbone: a 3-layer AWD-LSTM (Merity et al.) trained with fastai. - Starting point: Khooli's Arabic Wikipedia ULMFiT model. - LM fine-tuning: done on all available unlabelled task text, i.e. train plus dev. - Training schedule: gradual unfreezing. - Hardware: a Titan X GPU with batch size 64. - **Dev macro F1, subtask A:** 74 (forward), 78 (backward) and 80 (averaged). - **Dev macro F1, subtask B:** 60, 61 and 61, even though accuracy sits around 96. - **Test results:** - Subtask A: accuracy 86, macro F1 77. - Subtask B: accuracy 95, macro F1 58. - **Future work proposed:** morphology-aware Arabic tokenization, and using a language model to generate extra minority-class examples, alone or alongside random oversampling or SMOTE. ## Interesting ideas - Training the same architecture once reading left-to-right and once right-to-left, then averaging outputs, gave a nearly free improvement on the rare class; direction reversal is a cheap source of model diversity. - The paper shows how a 95–96% accuracy can coexist with a macro F1 near 60, so weighted or accuracy-style scores are misleading whenever the rare label is the one that matters. - Before training the classifier, the language model is first adapted to the raw text of the target domain, using unlabelled material that labelled training would otherwise waste. - Instead of discarding surface cues, the tokenizer turns repetition and capitalization into explicit marker tokens, so the model still sees emphasis without bloating the vocabulary. - The authors suggest using a generative language model to synthesize additional examples of the underrepresented class, as an alternative or complement to classic oversampling. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): a right-to-left variant of the perceptron parser or gparse could serve as an extra, partially independent witness when voting on silver trees. - **I2** (learn): evaluation reports should give macro and per-label scores for rare dependency relations, rare frames, and irony, so gains on frequent cases cannot mask failures on scarce ones. - **I3** (tokens): the tagger and lemma models could be refreshed on unlabelled fables, school texts, and word problems before supervised training, narrowing the gap between treebank genre and target genre. - **I4** (frames): keeping such emphasis markers as token features would give the irony and speech-act components an explainable signal (shouting, letter stretching) rather than losing it in normalization. - **I5** (learn): an LLM teacher tier could generate extra sentences for rare constructions or rare frames, which witness parsers would then vet before the sentences enter training. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works ## Cite Cite as: Mohamed Abdellatif and Ahmed Elgammal. 2020. Offensive language detection in Arabic using ULMFiT. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{abdellatif2020offensive, title = {{Offensive language detection in Arabic using ULMFiT}}, author = {Mohamed Abdellatif and Ahmed Elgammal}, booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection}, year = {2020}, url = {https://aclanthology.org/2020.osact-1.13/} } ``` - [← previous Multi-Task Sequence Prediction For Tunisian Arabizi Multi-Level Annotation](https://md.research.elamur.ai/free/en/papers/gugliotta-2020-multi-task-sequence-prediction-tunisian/) - [next → Text and Speech-based Tunisian Arabic Sub-Dialects Identification](https://md.research.elamur.ai/free/en/papers/abdallah-2020-text-speech-based-tunisian-arabic/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/9/) ## References 16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). Journal of machine learning research, 3(Feb):1137–1155, 2003. 2. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. [Smote: synthetic minority over-sampling technique](https://md.research.elamur.ai/free/en/papers/chawla-2002-smote-synthetic-minority-over-sampling/). Journal of artificial intelligence research, 16: 321–357, 2002. [↗](https://doi.org/10.1613/jair.953) 3. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018. [↗](https://arxiv.org/abs/1810.04805) 4. A. Ghazikhani, H. S. Yazdi, and R. Monsefi. Class imbalance handling using wrapper-based random oversampling. In 20th Iranian Conference on Electrical Engineering (ICEE2012), pages 611–616. IEEE, 2012. [find](https://scholar.google.com/scholar?q=A%2E+Ghazikhani%2C+H%2E+S%2E+Yazdi%2C+and+R%2E+Monsefi%2E+Class+imbalance+handling+using+wrapper%2Dbased+random+oversampling%2E+In+20th+Iranian+Conference+on+Electrical+Engineering+%28ICEE2012%29%2C+pages+611%E2%80%93616%2E+IEEE%2C+2012) 5. J. Howard and S. Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. [↗](https://arxiv.org/abs/1801.06146) 6. A. Khooli. Applied data science. [https://github.com/abedkhooli/ds2](https://github.com/abedkhooli/ds2), 2019. [↗](https://github.com/abedkhooli/ds2) 7. A. Mahendran and A. Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233–255, 2016. [↗](https://doi.org/10.1007/s11263-016-0911-8) 8. S. Merity, N. S. Keskar, and R. Socher. [Regularizing and optimizing lstm language models](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/). arXiv preprint arXiv:1708.02182, 2017. [↗](https://arxiv.org/abs/1708.02182) 9. H. Mubarak, K. Darwish, W. Magdy, T. Elsayed, and H. Al-Khalifa. [Overview of osact4 arabic offensive language detection shared task](https://md.research.elamur.ai/free/en/papers/mubarak-2020-overview-osact4-arabic-offensive-language/). 4, 2020. 10. Macro Precision Recall 75 57 77 58 86 57 F 1 94 95 95 F 1 60 61 61 Macro precision recall 79 76 75 56 F 1 77 58 J. H. Park and P. Fung. [One-step and two-step classification for abusive language detection on twitter](https://md.research.elamur.ai/free/en/papers/park-2017-one-step-two-step-classification/). arXiv preprint arXiv:1706.01206, 2017. [↗](https://arxiv.org/abs/1706.01206) 11. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018. [↗](https://s3-us-west-2.amazonaws) 12. S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf. [Transfer learning in natural language processing](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15–18, 2019. 13. A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014. [↗](https://doi.org/10.48550/arxiv.1403.6382) 14. L. N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464–472. [find](https://scholar.google.com/scholar?q=L%2E+N%2E+Smith%2E+Cyclical+learning+rates+for+training+neural+networks%2E+In+2017+IEEE+Winter+Conference+on+Applications+of+Computer+Vision+%28WACV%29%2C+pages+464%E2%80%93472%2E) 15. Z. Waseem and D. Hovy. [Hateful symbols or hateful people? predictive features for hate speech detection on twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/). In Proceedings of the NAACL student research workshop, pages 88–93, 2016. 16. Z. Zhang, D. Robinson, and J. Tepper. Detecting hate speech on twitter using a convolution-gru based deep neural network. In European semantic web conference, pages 745–760. Springer, 2018. [find](https://scholar.google.com/scholar?q=Z%2E+Zhang%2C+D%2E+Robinson%2C+and+J%2E+Tepper%2E+Detecting+hate+speech+on+twitter+using+a+convolution%2Dgru+based+deep+neural+network%2E+In+European+semantic+web+conference%2C+pages+745%E2%80%93760%2E+Springer%2C+2018%2E) *Updated 1 November 2026*