---
title: "Exposing the limits of Zero-shot Cross-lingual Hate Speech"
description: "This short paper asks whether hate speech classifiers trained on English (or another single language)… 43 references, cited by 61 papers here."
authors: ["Debora Nozza"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)"
source: "https://aclanthology.org/2021.acl-short.114/"
doi: "10.18653/v1/2021.acl-short.114"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nozza-2021-exposing-limits-zero-shot-cross/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nozza-2021-exposing-limits-zero-shot-cross/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection

# Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection

**Authors:** [Debora Nozza](https://md.research.elamur.ai/free/en/authors/debora-nozza/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-short.114/) · [PDF](https://aclanthology.org/2021.acl-short.114.pdf) · [DOI 10.18653/v1/2021.acl-short.114](https://doi.org/10.18653/v1/2021.acl-short.114) · [OpenAlex](https://openalex.org/W3174882277)

**Citations:** 66 citations (OpenAlex, 2026-10-14) · 61 in our library, of the 141,840 papers whose bibliographies we read

## Summary
This short paper asks whether hate speech classifiers trained on English (or another single language) can be applied directly to languages with no labeled data. Nozza fine-tunes mBERT, and later XLM-R, on Twitter benchmarks in English, Italian and Spanish: HatEval for English and Spanish, and the Evalita corpora AMI and HaSpeeDe for Italian. The two hate targets, immigrants and women, are kept separate. The study compares monolingual, zero-shot cross-lingual, few-shot cross-lingual and multi-language training. It uses Sampling and Occlusion (SOC) hierarchical explanations to see which phrases drive predictions.

Zero-shot transfer works reasonably well for anti-immigrant hate but fails for misogyny. The cause is common swear expressions such as *porca puttana*, *porca troia* and *puta*. They literally insult women but are usually just intensifiers. A model never exposed to them in the target language reads them literally and flags harmless tweets as misogynous. When both targets are merged, the averaged scores look acceptable and hide this failure. The paper concludes that zero-shot transfer is not a ready-made fix for low-resource hate speech detection.

## Key points
- **Data:**
  - HatEval (EN, ES) covers both targets.
  - For Italian, AMI covers women and HaSpeeDe covers immigrants.
  - Training sets range from about 1.6k to 4.5k tweets per language and target.
- **English immigrant baseline:** the monolingual English model for immigrants scored worst (macro-F1 0.368). It overfit to collection hashtags such as #SendThemBack. Removing them raised the score to 0.438.
- **Taboo interjections in the women test sets:**
  - They appear in 29% of the Italian and 78% of the Spanish test tweets.
  - On these tweets, zero-shot accuracy was 3% for IT and 58% for ES. Monolingual accuracy was 99% for IT and 82% for ES.
  - All 276 Italian *porca puttana* tweets are non-misogynous, and the zero-shot model got every one of them wrong.
- **Few-shot and XLM-R results:**
  - Adding just 1% of target-language training data gave macro-F1 0.660 for ES+EN⇒IT and 0.702 for EN+IT⇒ES.
  - XLM-R reaches 0.806 for IT⇒IT and 0.859 for ES⇒ES monolingually, but loses about 36% macro-F1 in zero-shot settings.
- **Merged targets:** zero-shot macro-F1 with both targets combined was 0.572 for ⇒EN, 0.513 for ⇒IT and 0.632 for ⇒ES. These middling numbers mask the misogyny breakdown.

## Interesting ideas
- **Report results per target or subcategory instead of pooled.** An averaged score can hide a subtask that collapses entirely. This applies to any multi-class or multi-domain benchmark.
- **Check failures with phrase-level explanations, then confirm them with an ablation.** The paper first used SOC to spot suspicious phrases. It then retrained without those phrases, or without the hashtags, to show the effect is causal. This turns an anecdote from an explanation tool into evidence.
- **Audit idioms and swear words that carry a literal meaning different from their usual use.** Words that do not translate one-to-one are where cross-lingual transfer breaks. For Ukrainian or Russian content, a curated list of such expressions would be a cheap diagnostic for any transferred classifier.
- **Treat a tiny in-language sample as a strong baseline.** Even 1% of target-language data recovered much of the lost performance. When planning a low-resource project, budgeting a small annotation effort may be worth more than relying on pure zero-shot transfer.
- **Watch for keyword-driven collection artifacts.** Hashtags or terms used to gather data can become shortcuts the model overfits to. Training on another language sometimes avoids them by accident, which is worth checking in any scraped corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Bocconi University](https://md.research.elamur.ai/free/en/universities/bocconi-university/) — University, Milan, Italy · 16 works · 2,850 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Debora Nozza. 2021. Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nozza2021exposing,
  title = {{Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection}},
  author = {Debora Nozza},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.114},
  url = {https://doi.org/10.18653/v1/2021.acl-short.114}
}
```

- [← previous Efficient Nearest Neighbor Language Models](https://md.research.elamur.ai/free/en/papers/he-2021-efficient-nearest-neighbor-language-models/)
- [next → Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics](https://md.research.elamur.ai/free/en/papers/bhargava-2021-generalization-nli-ways-not-go/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. [SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/). In Proceedings of the 13th International Wor…
2. Federico Bianchi and Dirk Hovy. 2021. On the gap between adoption and understanding in NLP. In Findings of the Association for Computational Linguistics: ACL 2021. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Federico+Bianchi+and+Dirk+Hovy%2E+2021%2E+On+the+gap+between+adoption+and+understanding+in+NLP%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2021%2E+Association+for+Computational+Lingui)
3. Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. 2021. [Cross-lingual contextualized topic models with zero-shot learning](https://md.research.elamur.ai/free/en/papers/bianchi-2021-cross-lingual-contextualized-topic-models/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1676–1683, O…
4. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. [Man is to computer programmer as woman is to homemaker? Debiasing word embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/). In Advances in neural information processing systems, pages 4349–4357.
5. Cristina Bosco, Dell’Orletta Felice, Fabio Poletto, Manuela Sanguinetti, and Tesconi Maurizio. 2018. [Overview of the EVALITA 2018 hate speech detection task](https://md.research.elamur.ai/free/en/papers/bosco-2018-overview-evalita-2018-hate-speech/). In Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.
6. Final Workshop (EVALITA 2018), volume 2263, pages 1–9, Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+volume+2263%2C+pages+1%E2%80%939%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
7. Tommaso Caselli, Nicole Novielli, Viviana Patti, and Paolo Rosso. 2018. [find](https://scholar.google.com/scholar?q=Tommaso+Caselli%2C+Nicole+Novielli%2C+Viviana+Patti%2C+and+Paolo+Rosso%2E+2018%2E)
8. EVALITA 2018: Overview of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. [find](https://scholar.google.com/scholar?q=EVALITA+2018%3A+Overview+of+the+6th+Evaluation+Campaign+of+Natural+Language+Processing+and+Speech+Tools+for+Italian%2E+In+Proceedings+of+Sixth+Evaluation+Campaign+of+Natural+Language+Processing+and+Speech)
9. Final Workshop (EVALITA 2018), Turin, Italy. CEUR.org. [find](https://scholar.google.com/scholar?q=Final+Workshop+%28EVALITA+2018%29%2C+Turin%2C+Italy%2E+CEUR%2Eorg%2E)
10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
11. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. [Automated hate speech detection and the problem of offensive language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/). In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17, pages 512–515. AAAI Press.
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
13. Ashwin Geet D’Sa, Irina Illina, Dominique Fohr, Dietrich Klakow, and Dana Ruiter. 2020. Label propagation-based semi-supervised learning for hate speech classification. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 54–59, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Ashwin+Geet+D%E2%80%99Sa%2C+Irina+Illina%2C+Dominique+Fohr%2C+Dietrich+Klakow%2C+and+Dana+Ruiter%2E+2020%2E+Label+propagation%2Dbased+semi%2Dsupervised+learning+for+hate+speech+classification%2E+In+Proceedings+of+the+First+Wor)
14. Elisabetta Fersini, Debora Nozza, and Giulia Boifava. 2020a. Profiling Italian misogynist: An empirical study. In Proceedings of the Workshop on Resources and Techniques for User and Author Profiling in Abusive Language, pages 9–13, Marseille, France. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Elisabetta+Fersini%2C+Debora+Nozza%2C+and+Giulia+Boifava%2E+2020a%2E+Profiling+Italian+misogynist%3A+An+empirical+study%2E+In+Proceedings+of+the+Workshop+on+Resources+and+Techniques+for+User+and+Author+Profiling+)
15. Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2018. [Overview of the EVALITA 2018 task on automatic misogyny identification (AMI)](https://md.research.elamur.ai/free/en/papers/fersini-2018-overview-evalita-2018-task-automatic/). Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2018), 12:59.
16. Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2020b. AMI @ EVALITA2020: Automatic misogyny identification. In Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020), Online. CEUR.org. [find](https://scholar.google.com/scholar?q=Elisabetta+Fersini%2C+Debora+Nozza%2C+and+Paolo+Rosso%2E+2020b%2E+AMI+%40+EVALITA2020%3A+Automatic+misogyny+identification%2E+In+Proceedings+of+the+7th+evaluation+campaign+of+Natural+Language+Processing+and+Speech+)
17. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput. Surv., 51(4). [find](https://scholar.google.com/scholar?q=Paula+Fortuna+and+Se%CC%81rgio+Nunes%2E+2018%2E+A+survey+on+automatic+detection+of+hate+speech+in+text%2E+ACM+Comput%2E+Surv%2E%2C+51%284%29%2E)
18. Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. [find](https://scholar.google.com/scholar?q=Antigoni+Founta%2C+Constantinos+Djouvas%2C+Despoina+Chatzakou%2C+Ilias+Leontiadis%2C+Jeremy+Blackburn%2C+Gianluca+Stringhini%2C+Athena+Vakali%2C+Michael+Sirivianos%2C+and+Nicolas+Kourtellis%2E+2018%2E+Large+scale+crowdso)
19. Eric Holgate, Isabel Cachola, Daniel Preoţiuc-Pietro, and Junyi Jessy Li. 2018. Why swear? analyzing and inferring the intentions of vulgar expressions. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4405–4414, Brussels, Belgium. Association for Com… [find](https://scholar.google.com/scholar?q=Eric+Holgate%2C+Isabel+Cachola%2C+Daniel+Preot%CC%A7iuc%2DPietro%2C+and+Junyi+Jessy+Li%2E+2018%2E+Why+swear%3F+analyzing+and+inferring+the+intentions+of+vulgar+expressions%2E+In+Proceedings+of+the+2018+Conference+on+Empir)
20. Dirk Hovy. 2018. The social and the neural network: How to make natural language processing about people again. In Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, pages 42–49, New Orleans, Louisiana, USA. Association for C… [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2E+2018%2E+The+social+and+the+neural+network%3A+How+to+make+natural+language+processing+about+people+again%2E+In+Proceedings+of+the+Second+Workshop+on+Computational+Modeling+of+People%E2%80%99s+Opinions%2C+Pe)
21. Dirk Hovy and Shannon L. Spruit. 2016. [The social impact of natural language processing](https://md.research.elamur.ai/free/en/papers/hovy-2016-social-impact-natural-language-processing/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics.
22. Dirk Hovy and Diyi Yang. 2021. [The importance of modeling social factors of language: Theory and practice](https://md.research.elamur.ai/free/en/papers/hovy-2021-importance-modeling-social-factors-language/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602, Online. Association for Computa…
23. Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Shrivastava, Nikhil Chakravartula, Manish Gupta, and Vasudeva Varma. 2019. FERMI at SemEval-2019 task 5: Using sentence embeddings to identify hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Sema… [find](https://scholar.google.com/scholar?q=Vijayasaradhi+Indurthi%2C+Bakhtiyar+Syed%2C+Manish+Shrivastava%2C+Nikhil+Chakravartula%2C+Manish+Gupta%2C+and+Vasudeva+Varma%2E+2019%2E+FERMI+at+SemEval%2D2019+task+5%3A+Using+sentence+embeddings+to+identify+hate+speec)
24. Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren. 2020. Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi… [find](https://scholar.google.com/scholar?q=Xisen+Jin%2C+Zhongyu+Wei%2C+Junyi+Du%2C+Xiangyang+Xue%2C+and+Xiang+Ren%2E+2020%2E+Towards+hierarchical+importance+attribution%3A+Explaining+compositional+semantics+for+neural+sequence+models%2E+In+8th+International+C)
25. Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020. [Contextualizing hate speech classifiers with post-hoc explanation](https://md.research.elamur.ai/free/en/papers/kennedy-2020-contextualizing-hate-speech-classifiers-post/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435–5442, Online. Association for Computa…
26. Ritesh Kumar, Atul Kr. Ojha, Marcos Zampieri, and Shervin Malmasi, editors. 2018. Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). Association for Computational Linguistics, Santa Fe, New Mexico, USA. [find](https://scholar.google.com/scholar?q=Ritesh+Kumar%2C+Atul+Kr%2E+Ojha%2C+Marcos+Zampieri%2C+and+Shervin+Malmasi%2C+editors%2E+2018%2E+Proceedings+of+the+First+Workshop+on+Trolling%2C+Aggression+and+Cyberbullying+%28TRAC%2D2018%29%2E+Association+for+Computational)
27. Sotiris Lamprinidis, Federico Bianchi, Daniel Hardt, and Dirk Hovy. 2021. Universal joy a data set and results for classifying emotions across languages. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 62–75, Online. Ass… [find](https://scholar.google.com/scholar?q=Sotiris+Lamprinidis%2C+Federico+Bianchi%2C+Daniel+Hardt%2C+and+Dirk+Hovy%2E+2021%2E+Universal+joy+a+data+set+and+results+for+classifying+emotions+across+languages%2E+In+Proceedings+of+the+Eleventh+Workshop+on+Com)
28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
29. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. [CrowS-pairs: A challenge dataset for measuring social biases in masked language models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
30. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making sense of language-specific BERT models. arXiv preprint arXiv:2003.02912. — [arXiv:2003.02912](https://arxiv.org/abs/2003.02912) [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Federico+Bianchi%2C+and+Dirk+Hovy%2E+2020%2E+What+the+%5BMASK%5D%3F+Making+sense+of+language%2Dspecific+BERT+models%2E+arXiv+preprint+arXiv%3A2003%2E02912%2E+%E2%80%94+%5BarXiv%3A2003%2E02912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E02)
31. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. [HONEST: Measuring hurtful sentence completion in language models](https://md.research.elamur.ai/free/en/papers/nozza-2021-honest-measuring-hurtful-sentence-completion/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406, Online. Associati…
32. Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. 2019. Unintended bias in misogyny detection. In IEEE/WIC/ACM International Conference on Web Intelligence, WI ’19, page 149–155, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Claudia+Volpetti%2C+and+Elisabetta+Fersini%2E+2019%2E+Unintended+bias+in+misogyny+detection%2E+In+IEEE%2FWIC%2FACM+International+Conference+on+Web+Intelligence%2C+WI+%E2%80%9919%2C+page+149%E2%80%93155%2C+New+York%2C+NY%2C+U)
33. Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. 2020. Misogyny detection in twitter: a multilingual and cross-domain study. Information Processing & Management, 57(6):102360. [find](https://scholar.google.com/scholar?q=Endang+Wahyu+Pamungkas%2C+Valerio+Basile%2C+and+Viviana+Patti%2E+2020%2E+Misogyny+detection+in+twitter%3A+a+multilingual+and+cross%2Ddomain+study%2E+Information+Processing+%26+Management%2C+57%286%29%3A102360%2E)
34. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics.
35. Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2020. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, pages 1–47. [find](https://scholar.google.com/scholar?q=Fabio+Poletto%2C+Valerio+Basile%2C+Manuela+Sanguinetti%2C+Cristina+Bosco%2C+and+Viviana+Patti%2E+2020%2E+Resources+and+benchmark+corpora+for+hate+speech+detection%3A+a+systematic+review%2E+Language+Resources+and+Eval)
36. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. [The woman worked as a babysitter: On biases in language generation](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process…
37. Hajung Sohn and Hyunju Lee. 2019. MC-BERT4HATE: hate speech detection using multi-channel BERT for different languages and translations. In 2019 International Conference on Data Mining Workshops, ICDM Workshops 2019, Beijing, China, November 8-11, 2019, pages 551–559. IEEE. [find](https://scholar.google.com/scholar?q=Hajung+Sohn+and+Hyunju+Lee%2E+2019%2E+MC%2DBERT4HATE%3A+hate+speech+detection+using+multi%2Dchannel+BERT+for+different+languages+and+translations%2E+In+2019+International+Conference+on+Data+Mining+Workshops%2C+ICDM)
38. Lukas Stappen, Fabian Brunn, and Björn W. Schuller. 2020. Cross-lingual zero- and few-shot hate speech detection utilising frozen transformer language models and AXEL. CoRR, abs/2004.13850. — [arXiv:2004.13850](https://arxiv.org/abs/2004.13850) [find](https://scholar.google.com/scholar?q=Lukas+Stappen%2C+Fabian+Brunn%2C+and+Bjo%CC%88rn+W%2E+Schuller%2E+2020%2E+Cross%2Dlingual+zero%2D+and+few%2Dshot+hate+speech+detection+utilising+frozen+transformer+language+models+and+AXEL%2E+CoRR%2C+abs%2F2004%2E13850%2E+%E2%80%94+%5BarXiv%3A)
39. Kenneth Steimel, Daniel Dakota, Yue Chen, and Sandra Kübler. 2019. Investigating multilingual abusive language detection: A cautionary tale. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1151–1160, Varna, Bulgaria. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Kenneth+Steimel%2C+Daniel+Dakota%2C+Yue+Chen%2C+and+Sandra+Ku%CC%88bler%2E+2019%2E+Investigating+multilingual+abusive+language+detection%3A+A+cautionary+tale%2E+In+Proceedings+of+the+International+Conference+on+Recent+A)
40. Bertie Vidgen and Leon Derczynski. 2021. Directions in abusive language training data, a systematic review: Garbage in, garbage out. PLOS ONE, 15(12):1–32. [find](https://scholar.google.com/scholar?q=Bertie+Vidgen+and+Leon+Derczynski%2E+2021%2E+Directions+in+abusive+language+training+data%2C+a+systematic+review%3A+Garbage+in%2C+garbage+out%2E+PLOS+ONE%2C+15%2812%29%3A1%E2%80%9332%2E)
41. Zeerak Waseem and Dirk Hovy. 2016. [Hateful symbols or hateful people? predictive features for hate speech detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/). In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California. Association for Computational Linguistics.
42. Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language. In Proceedings of GermEval 2018, 14th Conference on Natural Language Processing (KON-VENS 2018). [find](https://scholar.google.com/scholar?q=Michael+Wiegand%2C+Melanie+Siegel%2C+and+Josef+Ruppenhofer%2E+2018%2E+Overview+of+the+GermEval+2018+Shared+Task+on+the+Identification+of+Offensive+Language%2E+In+Proceedings+of+GermEval+2018%2C+14th+Conference+on)
43. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Proceedings of the Fourteenth Work… [find](https://scholar.google.com/scholar?q=Marcos+Zampieri%2C+Preslav+Nakov%2C+Sara+Rosenthal%2C+Pepa+Atanasova%2C+Georgi+Karadzhov%2C+Hamdy+Mubarak%2C+Leon+Derczynski%2C+Zeses+Pitenis%2C+and+C%CC%A7ag%CC%86r%C4%B1+C%CC%A7o%CC%88ltekin%2E+2020%2E+SemEval%2D2020+task+12%3A+Multilingual+offens)

*Updated 24 October 2026*