---
title: "GUCT at Arabic Hate Speech 2022: Towards a Better Isotropy for"
description: "They first show that MARBERT's contextual embedding space is highly anisotropic… 81.2% accuracy. 17 references, cited by 1 paper here."
authors: ["Nehal Elkaref", "Mervat Abu-Elkheir"]
year: 2022
venue: "Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection"
source: "https://aclanthology.org/2022.osact-1.27/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elkaref-2022-guct-arabic-hate-speech-2022/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elkaref-2022-guct-arabic-hate-speech-2022/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/8/) / GUCT at Arabic Hate Speech 2022: Towards a Better Isotropy for Hatespeech Detection

# GUCT at Arabic Hate Speech 2022: Towards a Better Isotropy for Hatespeech Detection

**Authors:** [Nehal Elkaref](https://md.research.elamur.ai/free/en/authors/nehal-elkaref/) and [Mervat Abu-Elkheir](https://md.research.elamur.ai/free/en/authors/mervat-abu-elkheir/) · **Year:** 2022 · **Venue:** [Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks 2022](https://md.research.elamur.ai/free/en/venues/osact/2022/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.osact-1.27/) · [PDF](https://aclanthology.org/2022.osact-1.27.pdf)

**Citations:** [1](https://md.research.elamur.ai/free/en/papers/elkaref-2022-guct-arabic-hate-speech-2022/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors took part in sub-task A of the OSACT 2022 shared task on Fine-grained Hate Speech Detection, a binary decision on whether a dialectal Arabic tweet is offensive. Tweets are short and give a language model little context, so the authors try to improve the input representations rather than the classifier. They first show that MARBERT's contextual embedding space is highly anisotropic: vectors crowd into a narrow cone, and rare words drift away from the origin. Its isotropy score is about 2.88e-08. They then apply the cluster-based post-processing of Rajaee and Pilehvar: K-means the vectors, centre each cluster, and project out a few dominant directions per cluster. This raises isotropy to roughly 0.97. Both raw and processed MARBERT vectors are fed to a biLSTM classifier. The isotropic version reaches 81.2% accuracy and 79.1% macro-F1 on the development set, against 77.2% F1 for raw vectors. On the test set it scores 76.5% accuracy and about 74.2% F1. The work suggests that cheap geometric post-processing of frozen embeddings can help downstream classification.

## Key points
- **Data:** about 8.5K training and 1.2K development tweets from the OSACT5 dataset (Mubarak et al., 2022). Mentions, URLs, diacritics, elongations and non-Arabic characters were removed.
- **Isotropy measurement:** isotropy is computed as the ratio of the minimum to the maximum of a partition function taken over the eigenvectors of the embedding matrix. MARBERT scores 2.88e-08. After processing, scores are 0.9564 with k=1, 0.9743 with k=10 and 0.9767 with k=20.
- **Setup:** 10 clusters, with 8 directions removed per cluster (embedding dimension divided by 100). The biLSTM has two hidden layers of size 102 and is trained with AdamW at a learning rate of 3e-5 for 10 epochs. The best checkpoint is kept by macro-F1.
- **Development results:** macro-F1 was 79.1% with isotropic vectors and 77.2% with raw vectors. The confusion matrix shows 766 of 898 non-offensive and 271 of 368 offensive tweets classified correctly.
- **Test results:** 76.5% accuracy. The paper reports F1 as 74.2% in some places and 74.5% in others.
- **Ablations:** k=2, 10 and 20 give nearly identical scores. Removing 16 or 30 directions also changes little, which the authors attribute either to losing useful semantics or to the scarcity of high-frequency words in tweets.

## Interesting ideas
- Before trusting a pretrained encoder, the authors compute a single isotropy number for its embedding space, giving a cheap diagnostic of representation degeneration.
- Instead of fine-tuning, they post-process frozen vectors by clustering them and removing each cluster's top principal directions, a training-free fix that restores spread.
- The authors note that an anisotropic space inflates cosine similarity between unrelated items, so raw similarity scores are misleading.
- They plot the embedding space with points coloured by word frequency, which visually exposes rare words clustering away from the origin.
- Their ablations sweep the number of clusters and directions and find a flat plateau, showing the gain is not tied to one fragile setting.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could log the isotropy of its transformer token vectors per layer, so that a degenerate encoder is flagged before parser training starts.
- **I2** (tree): this could be tried as a lightweight feature transform on gparse's encoder outputs before the arc and label scorers, with the change in LAS measured.
- **I3** (world): any embedding-based matching of mentions to WordNet/Wikidata ontology entries should centre the vectors and remove dominant directions first, so similarity reflects meaning rather than a shared offset.
- **I4** (learn): frequency-coloured projections of silver-data token vectors could reveal whether rare lemmas are poorly represented and need extra teacher-labelled examples.
- **I5** (learn): Mova's evaluation could routinely include such hyperparameter sweeps alongside negative controls, so that a reported improvement holds across a range of settings.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (3): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 972 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Nehal Elkaref and Mervat Abu-Elkheir. 2022. GUCT at Arabic Hate Speech 2022: Towards a Better Isotropy for Hatespeech Detection. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{elkaref2022guct,
  title = {{GUCT at Arabic Hate Speech 2022: Towards a Better Isotropy for Hatespeech Detection}},
  author = {Nehal Elkaref and Mervat Abu-Elkheir},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.27/}
}
```

- [← previous Frequency-centroid features for word recognition of non-native English speakers](https://md.research.elamur.ai/free/en/papers/berjon-2022-frequency-centroid-features-word-recognition/)
- [next → Harnessing Multilingual Resources to Question Answering in Arabic](https://md.research.elamur.ai/free/en/papers/alnajjar-2022-harnessing-multilingual-resources-question-answering/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/8/)

## References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2020). [Arbert & marbert: deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). arXiv preprint arXiv:2101.01785. [↗](https://arxiv.org/abs/2101.01785)
2. Abuzayed, A. and Elsayed, T. (2020). [Quick and simple approach for detecting hate speech in arabic tweets](https://md.research.elamur.ai/free/en/papers/abuzayed-2020-quick-simple-approach-detecting-hate/). In Proceedings of the 4th workshop on open-source Arabic Corpora and processing tools, with a shared task on offensive language detection, pages 109–114.
3. Arora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A. (2016). [A latent variable model approach to pmi-based word embeddings](https://md.research.elamur.ai/free/en/papers/arora-2016-latent-variable-model-approach-pmi/). Transactions of the Association for Computational Linguistics, 4:385–399. [↗](https://doi.org/10.1162/tacl_a_00106)
4. Church, K. W. (2017). Word2vec. Natural Language Engineering, 23(1):155–162. [find](https://scholar.google.com/scholar?q=Church%2C+K%2E+W%2E+%282017%29%2E+Word2vec%2E+Natural+Language+Engineering%2C+23%281%29%3A155%E2%80%93162%2E)
5. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
6. Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. (2019). [Representation degeneration problem in training natural language generation models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/). arXiv preprint arXiv:1907.12009. [↗](https://arxiv.org/abs/1907.12009)
7. Huang, L., Yang, D., Lang, B., and Deng, J. (2018). Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 791–800. [find](https://scholar.google.com/scholar?q=Huang%2C+L%2E%2C+Yang%2C+D%2E%2C+Lang%2C+B%2E%2C+and+Deng%2C+J%2E+%282018%29%2E+Decorrelated+batch+normalization%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+791%E2%80%93800%2E)
8. Husain, F. (2020). [Osact4 shared task on offensive language detection: Intensive preprocessing-based approach](https://md.research.elamur.ai/free/en/papers/husain-2020-osact4-shared-task-offensive-language/). arXiv preprint arXiv:2005.07297. [↗](https://arxiv.org/abs/2005.07297)
9. Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR. [↗](https://doi.org/10.48550/arxiv.1502.03167)
10. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T. (2016). [Fasttext](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/). zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. [↗](https://arxiv.org/abs/1612.03651)
11. Liang, Y., Cao, R., Zheng, J., Ren, J., and Gao, L. (2021). Learning to remove: Towards isotropic pre-trained bert embedding. In International Conference on Artificial Neural Networks, pages 448–459. Springer. https://github.com/nehalelkaref/Towards-a-Better-Isotropy-to-Detect-Hatespeech. git Mu, J., Bhat, S., and Viswanath, P. (2017). All-but-the-top: Simple and effective postprocessing for word representations. arXiv preprint arXiv:1702.01417. [↗](https://arxiv.org/abs/1702.01417)
12. Mubarak, H., Hassan, S., and Chowdhury, S. A. (2022). [Emojis as anchors to detect arabic offensive language and hate speech](https://md.research.elamur.ai/free/en/papers/mubarak-2022-emojis-anchors-detect-arabic-offensive/). arXiv preprint arXiv:2201.06723. [↗](https://arxiv.org/abs/2201.06723)
13. Rajaee, S. and Pilehvar, M. T. (2021a). A cluster-based approach for improving isotropy in contextual embedding space. arXiv preprint arXiv:2106.01183. [↗](https://arxiv.org/abs/2106.01183)
14. Rajaee, S. and Pilehvar, M. T. (2021b). An isotropy analysis in the multilingual bert embedding space. arXiv preprint arXiv:2110.04504. Rajaee, S. and Sheikhi, M. ). Isotropic contextual word representation in bert. [↗](https://arxiv.org/abs/2110.04504)
15. Saeed, H. H., Calders, T., and Kamiran, F. (2020). [Osact4 shared tasks: Ensembled stacked classification for offensive and hate speech in arabic tweets](https://md.research.elamur.ai/free/en/papers/saeed-2020-osact4-shared-tasks-ensembled-stacked/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 71–75.
16. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. [find](https://scholar.google.com/scholar?q=Soliman%2C+A%2E+B%2E%2C+Eissa%2C+K%2E%2C+and+El%2DBeltagy%2C+S%2E+R%2E+%282017%29%2E+Aravec%3A+A+set+of+arabic+word+embedding+models+for+use+in+arabic+nlp%2E+Procedia+Computer+Science%2C+117%3A256%E2%80%93265%2E)
17. Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., and Gu, Q. (2019). Improving neural language generation with spectrum control. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wang%2C+L%2E%2C+Huang%2C+J%2E%2C+Huang%2C+K%2E%2C+Hu%2C+Z%2E%2C+Wang%2C+G%2E%2C+and+Gu%2C+Q%2E+%282019%29%2E+Improving+neural+language+generation+with+spectrum+control%2E+In+International+Conference+on+Learning+Representations%2E)

*Updated 1 November 2026*