---
title: "Challenges for Toxic Comment Classification: An In-Depth Error"
description: "They train a range of shallow and neural models on two corpora: the multi-label Wikipedia talk page… 43 references, cited by 66 papers here."
authors: ["Betty van Aken", "Julian Risch", "Ralf Krestel", "Alexander Löser"]
year: 2018
venue: "Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)"
source: "https://aclanthology.org/W18-5105/"
doi: "10.18653/v1/w18-5105"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aken-2018-challenges-toxic-comment-classification-depth/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aken-2018-challenges-toxic-comment-classification-depth/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Challenges for Toxic Comment Classification: An In-Depth Error Analysis

# Challenges for Toxic Comment Classification: An In-Depth Error Analysis

**Authors:** [Betty van Aken](https://md.research.elamur.ai/free/en/authors/betty-van-aken/), Julian Risch, Ralf Krestel and [Alexander Löser](https://md.research.elamur.ai/free/en/authors/alexander-loser/) · **Year:** 2018 · **Venue:** [Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)](https://md.research.elamur.ai/free/en/venues/workshop-on-abusive-language-online/) ([Workshop on Abusive Language Online 2018](https://md.research.elamur.ai/free/en/venues/workshop-on-abusive-language-online/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W18-5105/) · [PDF](https://aclanthology.org/W18-5105.pdf) · [DOI 10.18653/v1/w18-5105](https://doi.org/10.18653/v1/w18-5105) · [OpenAlex](https://openalex.org/W2890059736)

**Citations:** 235 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/aken-2018-challenges-toxic-comment-classification-depth/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Automatic detection of toxic online comments still fails on many real-world cases. The authors ask which difficulties current classifiers already handle and which remain open. They train a range of shallow and neural models on two corpora: the multi-label Wikipedia talk page data that Google Jigsaw released for a Kaggle challenge, and the multi-class Twitter hate speech set of Davidson et al. (2017). The models are Logistic Regression over word and character n-grams, a CNN, bidirectional LSTM and GRU networks, and a GRU with attention, with the neural models using pretrained word embeddings. The models' predictions overlap only partly, so combining them into an ensemble raises macro-averaged F1. The gain is largest on rare classes and on heterogeneous data. The central contribution is a manual analysis of the ensemble's false negatives and false positives, sorted into error classes. It shows that all approaches share the same weaknesses:

- inconsistent annotation,
- rare or obfuscated vocabulary,
- harmless use of swear words,
- quoted abuse,
- sarcasm.

This points future work toward cleaner labels and representations that capture context and world knowledge.

## Key points
- The Wikipedia (Jigsaw/Kaggle) corpus has 223,549 comments with six overlapping labels: toxic, severe toxic, obscene, threat, insult and identity hate. Of these, 201,081 comments carry no label. The toxic class covers 9.6% of the data, and threat covers only 0.3%.
- The Twitter dataset of Davidson et al. (2017) has 24,783 tweets in three mutually exclusive classes: hate, offensive and neither. Offensive tweets dominate at 77.4% because the tweets were collected with Hatebase.org seed keywords.
- Pearson correlation between the models' predictions, together with their F1 scores, shows that each model tends to get different examples wrong. Mixing shallow learners with deep networks in the ensemble proved especially effective.
- Results for false negatives (Wikipedia / Twitter):
  - questionable labels: 23% / 9%
  - rare or unknown words: 30% / 43%
  - quotations of or references to toxic language: 17% / 8%
  - sarcasm or irony: 11% / none
- Results for false positives (Wikipedia / Twitter):
  - doubtful labels: 53% on Wikipedia (comments marked clean that are arguably toxic) / 10% on Twitter
  - swear words in non-toxic text: 60% / 77%
  - rare words: 8% / 17%

## Interesting ideas
- **Diverse ensembles:** Build the ensemble from architectures that fail in different ways, and check this by correlating their predictions. Picking members for complementary errors rather than raw accuracy suits any imbalanced classification task.
- **Error classes for false positives and negatives:** Sample misclassified items from both directions and tag them with error classes, which makes vague "model weaknesses" measurable. The same protocol could be reused to audit Ukrainian NLP classifiers.
- **Auditing gold labels:** Include "doubtful label" as an explicit error class. This shows how much apparent model error is really annotation noise, which argues for tracking annotator agreement and writing clear class definitions when building datasets.
- **Paradigmatic context:** The concept explains failures such as moderator remarks that quote abuse or self-deprecating swearing. It is a useful lens for any task where trigger words appear in harmless frames.
- **Cross-domain validation:** Test the same pipeline on a second dataset with a different domain and task format (multi-label vs. multi-class). This cheaply checks whether conclusions generalise beyond one corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Potsdam](https://md.research.elamur.ai/free/en/universities/university-of-potsdam/) — University, Potsdam, Germany · 36 works · 5,236 citations
- [Hasso Plattner Institute](https://md.research.elamur.ai/free/en/labs/hasso-plattner-institute/) — Research institute, Potsdam, Germany · 7 works · 4,985 citations
- [Berliner Hochschule für Technik](https://md.research.elamur.ai/free/en/universities/berliner-hochschule-fur-technik/) — University, Berlin, Germany · 5 works · 670 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works

🧭 Topics (1): [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Betty van Aken, Julian Risch, Ralf Krestel and Alexander Löser. 2018. Challenges for Toxic Comment Classification: An In-Depth Error Analysis. Proceedings of the 2nd Workshop on Abusive Language Online (ALW2).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aken2018challenges,
  title = {{Challenges for Toxic Comment Classification: An In-Depth Error Analysis}},
  author = {Betty {van Aken}, Julian Risch, Ralf Krestel and Alexander Löser},
  booktitle = {Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)},
  year = {2018},
  doi = {10.18653/v1/w18-5105},
  url = {https://doi.org/10.18653/v1/w18-5105}
}
```

- [← previous Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach](https://md.research.elamur.ai/free/en/papers/xu-2018-unpaired-sentiment-sentiment-translation-cycled/)
- [next → Learning Named Entity Tagger using Domain-Specific Dictionary](https://md.research.elamur.ai/free/en/papers/shang-2018-learning-named-entity-tagger-domain/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In WWW. [find](https://scholar.google.com/scholar?q=Pinkesh+Badjatiya%2C+Shashank+Gupta%2C+Manish+Gupta%2C+and+Vasudeva+Varma%2E+2017%2E+Deep+learning+for+hate+speech+detection+in+tweets%2E+In+WWW%2E)
2. Cynthia Van Hee, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, Guy De Pauw, Walter Daelemans, and Véronique Hoste. 2015. Detection and fine-grained classification of cyberbullying events. In RANLP. [find](https://scholar.google.com/scholar?q=Cynthia+Van+Hee%2C+Els+Lefever%2C+Ben+Verhoeven%2C+Julie+Mennes%2C+Bart+Desmet%2C+Guy+De+Pauw%2C+Walter+Daelemans%2C+and+Ve%CC%81ronique+Hoste%2E+2015%2E+Detection+and+fine%2Dgrained+classification+of+cyberbullying+events%2E+In)
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). TACL, 5:135–146.
4. Akshita Jha and Radhika Mamidi. 2017. When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data. In NLP+CSS@ACL. [find](https://scholar.google.com/scholar?q=Akshita+Jha+and+Radhika+Mamidi%2E+2017%2E+When+does+a+compliment+become+sexist%3F+analysis+and+classification+of+ambivalent+sexism+using+twitter+data%2E+In+NLP%2BCSS%40ACL%2E)
5. Pete Burnap and Matthew L. Williams. 2015. Cyber hate speech on twitter : An application of machine classification and statistical modeling for policy and decision making. volume 7, pages 223–242. [find](https://scholar.google.com/scholar?q=Pete+Burnap+and+Matthew+L%2E+Williams%2E+2015%2E+Cyber+hate+speech+on+twitter+%3A+An+application+of+machine+classification+and+statistical+modeling+for+policy+and+decision+making%2E+volume+7%2C+pages+223%E2%80%93242%2E)
6. George W. Kennedy, Andrew W. McCollough, Edward Dixon, A. M. Parra Bastidas, J. Mark Ryan, and Chris Loo. 2017. Hack harassment : Technology solutions to combat online harassment. In ALW1@ACL. [find](https://scholar.google.com/scholar?q=George+W%2E+Kennedy%2C+Andrew+W%2E+McCollough%2C+Edward+Dixon%2C+A%2E+M%2E+Parra+Bastidas%2C+J%2E+Mark+Ryan%2C+and+Chris+Loo%2E+2017%2E+Hack+harassment+%3A+Technology+solutions+to+combat+online+harassment%2E+In+ALW1%40ACL%2E)
7. Pete Burnap and Matthew L. Williams. 2016. Us and them: identifying cyber hate on twitter across multiple protected characteristics. EPJ Data Science, 5:1–15. [find](https://scholar.google.com/scholar?q=Pete+Burnap+and+Matthew+L%2E+Williams%2E+2016%2E+Us+and+them%3A+identifying+cyber+hate+on+twitter+across+multiple+protected+characteristics%2E+EPJ+Data+Science%2C+5%3A1%E2%80%9315%2E)
8. Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. SOCIALCOM-PASSAT, pages 71–80. [find](https://scholar.google.com/scholar?q=Ying+Chen%2C+Yilu+Zhou%2C+Sencun+Zhu%2C+and+Heng+Xu%2E+2012%2E+Detecting+offensive+language+in+social+media+to+protect+adolescent+online+safety%2E+SOCIALCOM%2DPASSAT%2C+pages+71%E2%80%9380%2E)
9. Yoon Kim. 2014. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). In EMNLP.
10. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. In AAAI. [find](https://scholar.google.com/scholar?q=Irene+Kwok+and+Yuzhou+Wang%2E+2013%2E+Locate+the+hate%3A+Detecting+tweets+against+blacks%2E+In+AAAI%2E)
11. Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013. Improving cyberbullying detection with user context. In ECIR. [find](https://scholar.google.com/scholar?q=Maral+Dadvar%2C+Dolf+Trieschnigg%2C+Roeland+Ordelman%2C+and+Franciska+de+Jong%2E+2013%2E+Improving+cyberbullying+detection+with+user+context%2E+In+ECIR%2E)
12. Yashar Mehdad and Joel R. Tetreault. 2016. Do characters abuse more than words? In SIGDIAL. [find](https://scholar.google.com/scholar?q=Yashar+Mehdad+and+Joel+R%2E+Tetreault%2E+2016%2E+Do+characters+abuse+more+than+words%3F+In+SIGDIAL%2E)
13. Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. [Automated hate speech detection and the problem of offensive language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/). In ICWSM.
14. Dennis Njagi, Z Zuping, Damien Hanyurwimfura, and Jun Long. 2015. A lexicon-based approach for hate speech detection. In International Journal of Multimedia and Ubiquitous Engineering, volume 10, pages 215–230. [find](https://scholar.google.com/scholar?q=Dennis+Njagi%2C+Z+Zuping%2C+Damien+Hanyurwimfura%2C+and+Jun+Long%2E+2015%2E+A+lexicon%2Dbased+approach+for+hate+speech+detection%2E+In+International+Journal+of+Multimedia+and+Ubiquitous+Engineering%2C+volume+10%2C+page)
15. Karthik Dinakar, Birago Jones, Catherine Havasi, Henry Lieberman, and Rosalind W. Picard. 2012. Common sense reasoning for detection, prevention, and mitigation of cyberbullying. TiiS, 2:18:1–18:30. [find](https://scholar.google.com/scholar?q=Karthik+Dinakar%2C+Birago+Jones%2C+Catherine+Havasi%2C+Henry+Lieberman%2C+and+Rosalind+W%2E+Picard%2E+2012%2E+Common+sense+reasoning+for+detection%2C+prevention%2C+and+mitigation+of+cyberbullying%2E+TiiS%2C+2%3A18%3A1%E2%80%9318%3A30%2E)
16. Chikashi Nobata, Joel R. Tetreault, Achint Oommen Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In WWW. [find](https://scholar.google.com/scholar?q=Chikashi+Nobata%2C+Joel+R%2E+Tetreault%2C+Achint+Oommen+Thomas%2C+Yashar+Mehdad%2C+and+Yi+Chang%2E+2016%2E+Abusive+language+detection+in+online+user+content%2E+In+WWW%2E)
17. Björn Gambäck and Utpal Kumar Sikdar. 2017. [Using convolutional neural networks to classify hate-speech](https://md.research.elamur.ai/free/en/papers/gamback-2017-convolutional-neural-networks-classify-hate/). In ALW1@ACL.
18. Ji Ho Park and Pascale Fung. 2017. [One-step and two-step classification for abusive language detection on twitter](https://md.research.elamur.ai/free/en/papers/park-2017-one-step-two-step-classification/). CoRR, abs/1706.01206.
19. Lei Gao and Ruihong Huang. 2017. Detecting online hate speech using context aware models. In RANLP. [find](https://scholar.google.com/scholar?q=Lei+Gao+and+Ruihong+Huang%2E+2017%2E+Detecting+online+hate+speech+using+context+aware+models%2E+In+RANLP%2E)
20. John Pavlopoulos, Prodromos Malakasiotis, and Ion Androutsopoulos. 2017. Deeper attention to abusive user content moderation. In EMNLP. [find](https://scholar.google.com/scholar?q=John+Pavlopoulos%2C+Prodromos+Malakasiotis%2C+and+Ion+Androutsopoulos%2E+2017%2E+Deeper+attention+to+abusive+user+content+moderation%2E+In+EMNLP%2E)
21. Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Aristidis G. Vrahatis, and Vassilis P. Plagianakos. 2018. Convolutional neural networks for toxic comment classification. In SETN. [find](https://scholar.google.com/scholar?q=Spiros+V%2E+Georgakopoulos%2C+Sotiris+K%2E+Tasoulis%2C+Aristidis+G%2E+Vrahatis%2C+and+Vassilis+P%2E+Plagianakos%2E+2018%2E+Convolutional+neural+networks+for+toxic+comment+classification%2E+In+SETN%2E)
22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In EMNLP.
23. Michal Ptaszynski, Juuso Kalevi Kristian Eronen, and Fumito Masui. 2017. Learning deep on cyberbullying is always better than brute force. In IJCAI. [find](https://scholar.google.com/scholar?q=Michal+Ptaszynski%2C+Juuso+Kalevi+Kristian+Eronen%2C+and+Fumito+Masui%2E+2017%2E+Learning+deep+on+cyberbullying+is+always+better+than+brute+force%2E+In+IJCAI%2E)
24. Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Dawei Yin and Brian D. Davison. 2009. Detection of harassment on web 2 . 0. In CAW2.0@WWW. [find](https://scholar.google.com/scholar?q=Jennifer+Golbeck%2C+Zahra+Ashktorab%2C+Rashad+O%2E+Banjo%2C+Alexandra+Berlinger%2C+Siddharth+Bhagwan%2C+Cody+Buntain%2C+Paul+Cheakalos%2C+Alicia+A%2E+Geller%2C+Quint+Dawei+Yin+and+Brian+D%2E+Davison%2E+2009%2E+Detection+of+har)
25. Jing Qian, Mai ElSherief, Elizabeth M. Belding-Royer, and William Yang Wang. 2018. Leveraging intra-user and inter-user representation learning for automated hate speech detection. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jing+Qian%2C+Mai+ElSherief%2C+Elizabeth+M%2E+Belding%2DRoyer%2C+and+William+Yang+Wang%2E+2018%2E+Leveraging+intra%2Duser+and+inter%2Duser+representation+learning+for+automated+hate+speech+detection%2E+In+NAACL%2DHLT%2E)
26. Ziqi Zhang and Lei Luo. 2018. Hate speech detection: A solved problem? the challenging case of long tail on twitter. CoRR, abs/1803.03662. [find](https://scholar.google.com/scholar?q=Ziqi+Zhang+and+Lei+Luo%2E+2018%2E+Hate+speech+detection%3A+A+solved+problem%3F+the+challenging+case+of+long+tail+on+twitter%2E+CoRR%2C+abs%2F1803%2E03662%2E)
27. Julian Risch and Ralf Krestel. 2018. Aggression identification using deep learning and data augmentation. In TRAC-1@COLING, pages 150–158. [find](https://scholar.google.com/scholar?q=Julian+Risch+and+Ralf+Krestel%2E+2018%2E+Aggression+identification+using+deep+learning+and+data+augmentation%2E+In+TRAC%2D1%40COLING%2C+pages+150%E2%80%93158%2E)
28. Ziqi Zhang, David Robinson, and Jonathan A. Tepper. 2018. Detecting hate speech on twitter using a convolution-gru based deep neural network. In ESWC. [find](https://scholar.google.com/scholar?q=Ziqi+Zhang%2C+David+Robinson%2C+and+Jonathan+A%2E+Tepper%2E+2018%2E+Detecting+hate+speech+on+twitter+using+a+convolution%2Dgru+based+deep+neural+network%2E+In+ESWC%2E)
29. David Robinson, Ziqi Zhang, and Jonathan Tepper. 2018. Hate speech detection on twitter: Feature engineering v.s. feature selection. In ESWC. [find](https://scholar.google.com/scholar?q=David+Robinson%2C+Ziqi+Zhang%2C+and+Jonathan+Tepper%2E+2018%2E+Hate+speech+detection+on+twitter%3A+Feature+engineering+v%2Es%2E+feature+selection%2E+In+ESWC%2E)
30. Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J. Miller, and Cornelia Caragea. 2016. Content-driven detection of cyberbullying on the instagram social network. In IJCAI. [find](https://scholar.google.com/scholar?q=Haoti+Zhong%2C+Hao+Li%2C+Anna+Cinzia+Squicciarini%2C+Sarah+Michele+Rajtmajer%2C+Christopher+Griffin%2C+David+J%2E+Miller%2C+and+Cornelia+Caragea%2E+2016%2E+Content%2Ddriven+detection+of+cyberbullying+on+the+instagram+soc)
31. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. CoRR, abs/1701.08118. [find](https://scholar.google.com/scholar?q=Bjo%CC%88rn+Ross%2C+Michael+Rist%2C+Guillermo+Carbonell%2C+Benjamin+Cabrera%2C+Nils+Kurowsky%2C+and+Michael+Wojatzki%2E+2016%2E+Measuring+the+reliability+of+hate+speech+annotations%3A+The+case+of+the+european+refugee+cris)
32. Steven Zimmerman, Udo Kruschwitz, and Chris Fox. 2018. Improving hate speech detection with deep learning ensembles. In LREC. [find](https://scholar.google.com/scholar?q=Steven+Zimmerman%2C+Udo+Kruschwitz%2C+and+Chris+Fox%2E+2018%2E+Improving+hate+speech+detection+with+deep+learning+ensembles%2E+In+LREC%2E)
33. Niloofar Safi Samghabadi, Suraj Maharjan, Alan Sprague, Raquel Diaz-Sprague, and Thamar Solorio. 2017. Detecting nastiness in social media. In ALW1@ACL. [find](https://scholar.google.com/scholar?q=Niloofar+Safi+Samghabadi%2C+Suraj+Maharjan%2C+Alan+Sprague%2C+Raquel+Diaz%2DSprague%2C+and+Thamar+Solorio%2E+2017%2E+Detecting+nastiness+in+social+media%2E+In+ALW1%40ACL%2E)
34. Anna Schmidt and Michael Wiegand. 2017. [A survey on hate speech detection using natural language processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/). In SocialNLP@EACL.
35. Sanjana Sharma, Saksham Agrawal, and Manish Shrivastava. 2018. Degree based classification of harmful speech using twitter data. CoRR, abs/1806.04197. [find](https://scholar.google.com/scholar?q=Sanjana+Sharma%2C+Saksham+Agrawal%2C+and+Manish+Shrivastava%2E+2018%2E+Degree+based+classification+of+harmful+speech+using+twitter+data%2E+CoRR%2C+abs%2F1806%2E04197%2E)
36. Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In ITASEC. [find](https://scholar.google.com/scholar?q=Fabio+Del+Vigna%2C+Andrea+Cimino%2C+Felice+Dell%E2%80%99Orletta%2C+Marinella+Petrocchi%2C+and+Maurizio+Tesconi%2E+2017%2E+Hate+me%2C+hate+me+not%3A+Hate+speech+detection+on+facebook%2E+In+ITASEC%2E)
37. W. Lloyd Warner and Julia Hirschberg. 2012. [Detecting hate speech on the world wide web](https://md.research.elamur.ai/free/en/papers/warner-2012-detecting-hate-speech-world-wide/). In LSM@ACL.
38. Zeerak Waseem. 2016. [Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/). In NLP+CSS@EMNLP.
39. Zeerak Waseem and Dirk Hovy. 2016. [Hateful symbols or hateful people? predictive features for hate speech detection on twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/). In SRW@NAACL-HLT.
40. Zeerak Waseem, James Thorne, and Joachim Bingel. 2018. Bridging the gaps: Multi task learning for domain transfer of hate speech detection. Online Harassment, pages 29–55. [find](https://scholar.google.com/scholar?q=Zeerak+Waseem%2C+James+Thorne%2C+and+Joachim+Bingel%2E+2018%2E+Bridging+the+gaps%3A+Multi+task+learning+for+domain+transfer+of+hate+speech+detection%2E+Online+Harassment%2C+pages+29%E2%80%9355%2E)
41. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In WWW. [find](https://scholar.google.com/scholar?q=Ellery+Wulczyn%2C+Nithum+Thain%2C+and+Lucas+Dixon%2E+2017%2E+Ex+machina%3A+Personal+attacks+seen+at+scale%2E+In+WWW%2E)
42. Guang Xiang, Bin Fan, Ling Wang, Jason I. Hong, and Carolyn Penstein Rosé. 2012. Detecting offensive tweets via topical feature discovery over a large scale twitter corpus. In CIKM. [find](https://scholar.google.com/scholar?q=Guang+Xiang%2C+Bin+Fan%2C+Ling+Wang%2C+Jason+I%2E+Hong%2C+and+Carolyn+Penstein+Rose%CC%81%2E+2012%2E+Detecting+offensive+tweets+via+topical+feature+discovery+over+a+large+scale+twitter+corpus%2E+In+CIKM%2E)
43. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. [Hierarchical attention networks for document classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In NAACL-HLT.

*Updated 24 October 2026*