---
title: "Detecting Anti-Semitic Hate Speech using — Liu 2024"
description: "The paper examines automatic detection of anti-Semitic hate speech on Twitter."
authors: ["Dengyi Liu", "Minghao Wang", "Andrew G. Catlin"]
year: 2024
venue: "arXiv:2405.03794"
source: "https://arxiv.org/abs/2405.03794"
doi: "10.48550/arxiv.2405.03794"
arxiv: "2405.03794"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-detecting-anti-semitic-hate-speech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-detecting-anti-semitic-hate-speech/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Detecting Anti-Semitic Hate Speech using Transformer-based Large Language Models

# Detecting Anti-Semitic Hate Speech using Transformer-based Large Language Models

**Authors:** Dengyi Liu, Minghao Wang, Andrew G. Catlin · **Year:** 2024 · **Venue:** arXiv:2405.03794 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.03794) · [PDF](https://arxiv.org/pdf/2405.03794) · [DOI 10.48550/arxiv.2405.03794](https://doi.org/10.48550/arxiv.2405.03794) · [arXiv 2405.03794](https://arxiv.org/abs/2405.03794) · [OpenAlex](https://openalex.org/W6941216174)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper examines automatic detection of anti-Semitic hate speech on Twitter. The authors note that general chat models are poorly suited to such sensitive moderation and that online hate shifts quickly. They gathered about 10,000 Jewish-related tweets through the Twitter API. Two researchers labeled a 3,000-post subset independently, and labels were kept based on their agreement. The authors then compared two families of methods. The first combines traditional learners (SVM, Naive Bayes, Random Forests, k-NN and Logistic Regression) with several text representations: CountVectorizer, TfidfVectorizer, HashingVectorizer, Word2Vec and GloVe. The second fine-tunes transformers: BERT, DistilBERT, RoBERTa and LLaMA-2 (7B), with RoBERTa and LLaMA-2 also tried with LoRA. The transformers clearly outperform the classical pipelines, mainly because they recover many more hateful posts, while accuracy looks similar across most systems. LoRA reaches comparable quality in a fraction of the fine-tuning time. The study is a proof of concept for transformer-based moderation of a narrow hate category. It also flags compute cost and opacity as open problems.

## Key points
- Data: roughly 10,000 tweets collected by keyword and context, with 3,000 annotated by two independent reviewers who resolved labels by agreement.
- Classical models reach about 0.91 accuracy but recall of only about 0.55–0.67. Their F1 peaks at 0.77 (SVM with TF-IDF or hashing features), and Word2Vec-based pipelines are often very weak (SVM+Word2Vec F1 0.00).
- Transformers reach F1 of 0.79–0.83. BERT has the top accuracy (0.94, F1 0.81), and LLaMA-2 (7B)+LoRA has the top F1 in the table (0.83, recall 0.79). The text, however, credits BERT with the best F1.
- Fine-tuning costs were 176 min for BERT (30 epochs), 79 min for DistilBERT and 54 min for RoBERTa, all on an A4000. RoBERTa+LoRA took 21 min. LLaMA-2 took 112 min on an A100, versus 35 min with LoRA.
- The stated limitations are GPU and expertise requirements and black-box predictions that are hard to explain.

## Interesting ideas
- The authors cross every classical classifier with every feature representation, which separates the effect of the representation from the effect of the learner; such a grid quickly shows which component actually matters. — For Mova (learn): Evaluate parser and tagger changes as a grid over feature sets and learners, so each gain can be traced to one component instead of a bundled change.
- The paper reports epochs, GPU type and wall-clock minutes next to quality, which shows LoRA cutting training time about threefold for nearly the same F1; cost becomes part of the comparison. — For Mova (tree): Track training time and hardware alongside LAS for gparse runs, and test low-rank adapters for treebank or dialect variants instead of full retraining.
- Two annotators label each post independently, and only agreed labels are trusted, which reduces individual bias in a subjective task. — For Mova (learn): This mirrors the witness-parser idea, so items the witnesses disagree on can be routed to an LLM teacher tier or human review instead of entering silver data.
- On skewed data, near-identical accuracy (around 0.91) hides large gaps in recall, so the authors judge models mainly by recall and F1; accuracy alone would make weak classifiers look competitive. — For Mova (answer): Score the readers on school tests and fable QA with per-class precision and recall, especially for rare answer types, so a model that just guesses the majority answer is exposed.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2024detecting,
  title = {{Detecting Anti-Semitic Hate Speech using Transformer-based Large Language Models}},
  author = {Liu, Dengyi and Wang, Minghao and Catlin, Andrew G.},
  year = {2024},
  doi = {10.48550/arxiv.2405.03794},
  eprint = {2405.03794},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.03794}
}
```

- [← previous Capturing the full potential of Maltese language learning through ChatGPT](https://md.research.elamur.ai/free/en/papers/zammit-2024-capturing-full-potential-maltese-language/)
- [next → Developing Arabic Grammar Materials Based on Contextual Learning at Islamic Boarding School in Indonesia](https://md.research.elamur.ai/free/en/papers/hamidah-2024-developing-arabic-grammar-materials-based/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805, 2019. 1, 3, 4 [↗](https://arxiv.org/abs/1810.04805)
2. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. https://arxiv.org/abs/1910.01108, 2020. 1, 3, 4 [↗](https://arxiv.org/abs/1910.01108)
3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. https://arxiv.org/abs/1907.11692, 2019. 1, 3, 4 [↗](https://arxiv.org/abs/1907.11692)
4. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. https://arxiv.org/abs/2307.09288, 2023. 1, 3, 4 [↗](https://arxiv.org/abs/2307.09288)
5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv:2106.09685, 2021. 1, 3, 4 [↗](https://arxiv.org/abs/2106.09685)
6. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546. 2 [↗](https://arxiv.org/abs/1310.4546)
7. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv Preprint arXiv:1802.05365. 2 [↗](https://arxiv.org/abs/1802.05365)
8. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in python, 2011. 2 [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
9. William Warner and Julia Hirschberg. Detecting hate speech on the world wide web. Association for Computational Linguistics, Proceedings of the Second Workshop on Language in Social Media, 2012. 2 [find](https://scholar.google.com/scholar?q=William+Warner+and+Julia+Hirschberg%2E+Detecting+hate+speech+on+the+world+wide+web%2E+Association+for+Computational+Linguistics%2C+Proceedings+of+the+Second+Workshop+on+Language+in+Social+Media%2C+2012%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/warner-2012-detecting-hate-speech-world-wide/)
10. Thomas Davison, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. International AAAI Conference on Web and Social Media, 11, 2017. 2 [find](https://scholar.google.com/scholar?q=Thomas+Davison%2C+Dana+Warmsley%2C+Michael+Macy%2C+and+Ingmar+Weber%2E+Automated+hate+speech+detection+and+the+problem+of+offensive+language%2E+International+AAAI+Conference+on+Web+and+Social+Media%2C+11%2C+2017%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)
11. Hugo Haddad, Hala Mulki, Abdelmajid Oueslati, and Mourad Gridach. Benchmarking multi-class classification algorithms for automatic hate speech detection. Procedia Computer Science, 159:685–694, 2019. 2 [find](https://scholar.google.com/scholar?q=Hugo+Haddad%2C+Hala+Mulki%2C+Abdelmajid+Oueslati%2C+and+Mourad+Gridach%2E+Benchmarking+multi%2Dclass+classification+algorithms+for+automatic+hate+speech+detection%2E+Procedia+Computer+Science%2C+159%3A685%E2%80%93694%2C+2019%2E+)
12. Ziqi Zhang, Derek Robinson, and Jonathan Tepper. Detecting hate speech on twitter using a convolution-gru based deep neural network. In European Semantic Web Conference, pages 745–760. Springer, 2018. 2 [find](https://scholar.google.com/scholar?q=Ziqi+Zhang%2C+Derek+Robinson%2C+and+Jonathan+Tepper%2E+Detecting+hate+speech+on+twitter+using+a+convolution%2Dgru+based+deep+neural+network%2E+In+European+Semantic+Web+Conference%2C+pages+745%E2%80%93760%2E+Springer%2C+2018%2E)
13. Pete Burnap and Matthew L Williams. Us and them: identifying cyber hate on twitter across multiple protected characteristics. EPJ Data Science, 5(1):11, 2016. 2 [find](https://scholar.google.com/scholar?q=Pete+Burnap+and+Matthew+L+Williams%2E+Us+and+them%3A+identifying+cyber+hate+on+twitter+across+multiple+protected+characteristics%2E+EPJ+Data+Science%2C+5%281%29%3A11%2C+2016%2E+2)
14. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 2 [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E+2)
15. Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014. 2 [↗](https://arxiv.org/abs/1408.5882) [our card](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
16. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. 2 [↗](https://arxiv.org/abs/1406.1078)
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017. 2 [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+NeurIPS%2C+2017%2E+2)
18. Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. A bert-based transfer learning approach for hate speech detection in online social media. arXiv preprint arXiv:1910.12574, 2019. 3 [↗](https://arxiv.org/abs/1910.12574)
19. Lanqin Yuan, Tianyu Wang, Gabriela Ferraro, Hanna Suominen, and Marian-Andrei Rizoiu. Transfer learning for hate speech detection in social media. arXiv preprint arXiv:1906.03829, 2019. 3 [↗](https://arxiv.org/abs/1906.03829)
20. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019. 3 [↗](https://arxiv.org/abs/1909.10351)