---
title: "LISAC FSDM-USMBA Team at SemEval-2020 Task 12:"
description: "This paper presents the winning system for the Arabic track of OffensEval 2020… 93.9% accuracy. 7 references, cited by 7 papers here."
authors: ["Hamza Alami", "Said Ouatik El Alaoui", "Abdessamad Benlahbib", "Noureddine En-nahnahi"]
year: 2020
venue: "Proceedings of the Fourteenth Workshop on Semantic Evaluation"
source: "https://aclanthology.org/2020.semeval-1.275/"
doi: "10.18653/v1/2020.semeval-1.275"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alami-2020-lisac-fsdm-usmba-team-semeval/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alami-2020-lisac-fsdm-usmba-team-semeval/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/) / LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT's pretrain-finetune discrepancy for Arabic offensive language identification

# LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT's pretrain-finetune discrepancy for Arabic offensive language identification

**Authors:** [Hamza Alami](https://md.research.elamur.ai/free/en/authors/hamza-alami/), [Said Ouatik El Alaoui](https://md.research.elamur.ai/free/en/authors/said-ouatik-el-alaoui/), [Abdessamad Benlahbib](https://md.research.elamur.ai/free/en/authors/abdessamad-benlahbib/) and Noureddine En-nahnahi · **Year:** 2020 · **Venue:** [Proceedings of the Fourteenth Workshop on Semantic Evaluation](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2020](https://md.research.elamur.ai/free/en/venues/semeval/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.semeval-1.275/) · [PDF](https://aclanthology.org/2020.semeval-1.275.pdf) · [DOI 10.18653/v1/2020.semeval-1.275](https://doi.org/10.18653/v1/2020.semeval-1.275) · [OpenAlex](https://openalex.org/W3116878825)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/alami-2020-lisac-fsdm-usmba-team-semeval/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper presents the winning system for the Arabic track of OffensEval 2020 (SemEval-2020 Task 12, sub-task A). The task is a binary decision: is a tweet offensive (OFF) or not (NOT)? The authors fine-tune AraBERT, a BERT BASE-sized model pretrained on Arabic text. Their main contribution is how they treat emojis.

In the tweet itself, each emoji becomes the special token [MASK]. The emoji's English description is translated into Arabic and placed in a second input segment, separated by [SEP]. The motivation is that [MASK] appears during pretraining but almost never in task data. Keeping it in the input during fine-tuning and inference narrows this pretrain-finetune mismatch.

Text is segmented with Farasa before AraBERT tokenization. Token embeddings are max-pooled into a single tweet vector, which a sigmoid output layer classifies.

The authors compared three preprocessing variants. The [MASK]-plus-side-segment version was the strongest, and it reached 90.17% macro F1 on the hidden test set, ranking first. The work shows that cheap, pretraining-aware input design can measurably improve a fine-tuned transformer.

## Key points
- **Dataset:** the Arabic data comes from Mubarak et al. (2020) and contains 10,000 labelled tweets.
  - Split: 7,000 train, 1,000 dev, 2,000 test.
  - NOT tweets outnumber OFF tweets roughly two to one.
- **Model:**
  - AraBERT with the BERT BASE configuration: 12 layers, 768 hidden units, 12 heads, about 110M parameters.
  - Pretrained on about 70M Arabic sentences (~24GB of text).
  - Fine-tuned with max sequence length 128, 5 epochs, batch size 32, Adam at 1e-4 with warmup and weight decay.
  - Built with TensorFlow 2.0 and keras-bert.
- **Variants compared (Table 1):**
  - Vanilla, with no emoji handling: 88.82% macro F1.
  - AraBERTEmojisIN, with emojis replaced inline by their Arabic meanings: 88.42%.
  - AraBERTEmojisOUT, with [MASK] inline plus a separate meanings segment: 89.06% macro F1, 93.9% accuracy, and the best AUC.
- **Official result:** AraBERTEmojisOUT scored 90.17% macro F1 on test, ranking first.
  - Per class, NOT reached about 96% precision and recall.
  - OFF reached 83% precision and 85% recall; the authors attribute the gap to class imbalance.
- **Representation:** global max pooling over all token vectors, not the [CLS] vector, feeding a sigmoid trained with binary cross-entropy.

## Interesting ideas
- Instead of feeding a pretrained encoder symbols it has rarely seen, swap them for a token it learned during pretraining ([MASK]), so fine-tuning inputs look more like pretraining inputs.
- Keep the main sentence clean by moving extra information (here, emoji glosses) into a separate input segment rather than splicing it inline; inline insertion actually scored slightly worse than doing nothing.
- Turning an emoji into a natural-language description in the working language makes its emotional or evaluative content readable by an ordinary text model.
- The evaluation holds the model fixed and varies only preprocessing across three variants, so any gain can be credited to the input design itself.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could mask emojis, URLs or odd symbols with a known placeholder before encoding, so the transformer parser is not confused by unfamiliar subword pieces.
- **I2** (tokens): the tokenizer could emit a single placeholder token for an emoji or symbol in the sentence stream and attach its gloss as side metadata, so the UD tree is not distorted by injected words.
- **I3** (frames): prag could read emoji glosses as explicit cues for speech act, sentiment or irony detection, keeping the cue traceable rather than hidden in embeddings.
- **I4** (learn): when testing a tokenizer or normalization change, train otherwise identical models that differ in that one step, and keep the "no change" variant as a built-in control.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Hamza Alami, Said Ouatik El Alaoui, Abdessamad Benlahbib and Noureddine En-nahnahi. 2020. LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT's pretrain-finetune discrepancy for Arabic offensive language identification. Proceedings of the Fourteenth Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{alami2020lisac,
  title = {{LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT's pretrain-finetune discrepancy for Arabic offensive language identification}},
  author = {Hamza Alami, Said Ouatik {El Alaoui}, Abdessamad Benlahbib and Noureddine En-nahnahi},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.275},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.275}
}
```

- [← previous Incorporating Syntax and Frame Semantics in Neural Network for Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/guo-2020-incorporating-syntax-frame-semantics-neural/)
- [next → Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model](https://md.research.elamur.ai/free/en/papers/darwish-2020-arabic-diacritic-recovery-using-feature/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/)

## References

7 references, 6 of them with a card of ours. ↗ leads to the work.

1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. [Farasa: A fast and furious segmenter for arabic](https://md.research.elamur.ai/free/en/papers/abdelali-2016-farasa-fast-furious-segmenter-arabic/). In Proceedings of the Demonstrations Session, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 11–16. The Association for Computational Linguistics.
2. Wissam Antoun, Fady Baly, and Hazem M. Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). CoRR, abs/2003.00104. [↗](https://arxiv.org/abs/2003.00104)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
4. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. [Arabic offensive language on twitter: Analysis and experiments](https://md.research.elamur.ai/free/en/papers/mubarak-2021-arabic-offensive-language-twitter-analysis/). arXiv preprint arXiv:2004.02192. [↗](https://arxiv.org/abs/2004.02192)
5. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 5754–5764. [↗](https://doi.org/10.48550/arxiv.1906.08237)
6. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. [Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval)](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/). In Jonathan May, Ekaterina Shutova, Aurélie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Mohammad, editors, Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2019, Minneapolis, MN, USA, June 6-7, 2019, pages 75–86. Association for Computational Linguistics.
7. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of SemEval. [↗](https://doi.org/10.48550/arxiv.2006.07235)

*Updated 2 November 2026*