---
title: "Toxicity Detection — Pavlopoulos 2020"
description: "Most toxicity datasets and classifiers judge each comment in isolation, even though a reply's meaning often depends on what it answers."
authors: ["John Pavlopoulos", "Jeffrey Sorensen", "Lucas Dixon", "Nithum Thain", "Ion Androutsopoulos"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.396/"
doi: "10.18653/v1/2020.acl-main.396"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pavlopoulos-2020-toxicity-detection-does-context-really/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pavlopoulos-2020-toxicity-detection-does-context-really/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Toxicity Detection: Does Context Really Matter?

# Toxicity Detection: Does Context Really Matter?

**Authors:** [John Pavlopoulos](https://md.research.elamur.ai/free/en/authors/john-pavlopoulos/), [Jeffrey Sorensen](https://md.research.elamur.ai/free/en/authors/jeffrey-sorensen/), [Lucas Dixon](https://md.research.elamur.ai/free/en/authors/lucas-dixon/), [Nithum Thain](https://md.research.elamur.ai/free/en/authors/nithum-thain/), [Ion Androutsopoulos](https://md.research.elamur.ai/free/en/authors/ion-androutsopoulos/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.396/) · [PDF](https://aclanthology.org/2020.acl-main.396.pdf) · [DOI 10.18653/v1/2020.acl-main.396](https://doi.org/10.18653/v1/2020.acl-main.396) · [OpenAlex](https://openalex.org/W3032574806)

**Citations:** 117 citations (openalex-snapshot · 2026-10-14) · [81](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most toxicity datasets and classifiers judge each comment in isolation, even though a reply's meaning often depends on what it answers. The authors examine this assumption with Wikipedia Talk Pages conversations, where context means only the parent comment and the discussion title. They ask two questions: how often context changes human toxicity judgements, and whether context-aware classifiers detect toxicity better.

For the first question, they ran an AB-style crowdsourcing study on 250 comments. One annotator group saw context and the other did not. Context changed the label for a small but statistically significant share of comments, and it could make a post seem either more or less toxic.

For the second question, they built a larger corpus of 20k comments, half labelled with context and half without. They trained BILSTM and BERT models with several ways of using the parent comment, and also tested PERSPECTIVE and BERT-CCTK. No approach gave a consistent or significant gain from context. The likely reason is that context-sensitive comments are rare. The authors argue for larger datasets annotated in context and release their data and code.

## Key points
- **CAT-SMALL (250 comments):** 5.2% of comments received opposite labels depending on whether context was shown. Context increased perceived toxicity in about 3.6% of cases and decreased it in about 1.6%.
- **CAT-LARGE (20k comments):** CAT-LARGE-N holds 10k comments labelled without context, and CAT-LARGE-C holds 10k labelled with context. All models were tested on held-out in-context labels, using five-fold Monte Carlo cross-validation and ROC AUC.
- **Context-aware variants tested:** CA-BILSTM-BILSTM, CA-BILSTM-BERT, CA-SEP-BERT (parent and target joined with [SEP]) and concatenation-based CA-CONC models. Training on in-context labels (@C) versus out-of-context labels (@N) also made no consistent difference.
- **Best scores:** The externally trained systems performed best. PERSPECTIVE scored 79.27 and CA-CONC-PERSPECTIVE 81.89. BERT-CCTK scored 78.08 and CA-CONC-BERT-CCTK 81.69. These differences were not statistically significant.
- **Limitations and resources:** Context covered only the previous post and the title, and the comments were randomly sampled. Data and code are available at github.com/ipavlopoulos/context_toxicity.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 78 not yet.

- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://md.research.elamur.ai/free/en/papers/lin-2023-toxicchat-unveiling-hidden-challenges-toxicity/)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://md.research.elamur.ai/free/en/papers/markov-2022-holistic-approach-undesired-content-detection/)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pavlopoulos2020toxicity,
  title = {{Toxicity Detection: Does Context Really Matter?}},
  author = {Pavlopoulos, John and Sorensen, Jeffrey and Dixon, Lucas and Thain, Nithum and Androutsopoulos, Ion},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.396},
  url = {https://doi.org/10.18653/v1/2020.acl-main.396}
}
```

- [← previous Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets](https://md.research.elamur.ai/free/en/papers/fortuna-2020-toxic-hateful-offensive-or-abusive/)
- [next → TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking](https://md.research.elamur.ai/free/en/papers/wang-2020-tplinker-single-stage-joint-extraction/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. B. van Aken, J. Risch, R. Krestel, and A. Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. In 2nd Workshop on Abusive Language Online, pages 33–42, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=B%2E+van+Aken%2C+J%2E+Risch%2C+R%2E+Krestel%2C+and+A%2E+Lo%CC%88ser%2E+2018%2E+Challenges+for+toxic+comment+classification%3A+An+in%2Ddepth+error+analysis%2E+In+2nd+Workshop+on+Abusive+Language+Online%2C+pages+33%E2%80%9342%2C+Brussels%2C+Belg)
2. D. M. Blei, A. Y Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993–1022. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Blei%2C+A%2E+Y+Ng%2C+and+M%2E+I%2E+Jordan%2E+2003%2E+Latent+dirichlet+allocation%2E+Journal+of+Machine+Learning+Research%2C+3%28Jan%29%3A993%E2%80%931022%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
3. D. Borkan, L. Dixon, J. Sorensen, N. Thain, and L. Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 491–500. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=D%2E+Borkan%2C+L%2E+Dixon%2C+J%2E+Sorensen%2C+N%2E+Thain%2C+and+L%2E+Vasserman%2E+2019%2E+Nuanced+metrics+for+measuring+unintended+bias+with+real+data+for+text+classification%2E+In+Companion+Proceedings+of+The+2019+World+Wid)
4. The discussion title was used only by the human annotators that examined context, not by context-aware systems, which considered only the parent comment. [find](https://scholar.google.com/scholar?q=The+discussion+title+was+used+only+by+the+human+annotators+that+examined+context%2C+not+by+context%2Daware+systems%2C+which+considered+only+the+parent+comment%2E)
5. T. Davidson, D. Warmsley, M. Macy, and I. Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM, pages 512–515, Montreal, Canada. [find](https://scholar.google.com/scholar?q=T%2E+Davidson%2C+D%2E+Warmsley%2C+M%2E+Macy%2C+and+I%2E+Weber%2E+2017%2E+Automated+hate+speech+detection+and+the+problem+of+offensive+language%2E+In+ICWSM%2C+pages+512%E2%80%93515%2C+Montreal%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)
6. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, page 4171–4186, Minneapolis,MN,USA. [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2E%2DW%2E+Chang%2C+K%2E+Lee%2C+and+K%2E+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2DHLT%2C+page+4171%E2%80%934186%2C+Minneapolis%2CMN%2CUSA%2E)
7. N. Djuric, J. Zhou, R. Morris, M. Grbovic, V. Radosavljevic, and N. Bhamidipati. 2015. Hate speech detection with comment embeddings. In ICWWW, pages 29–30. [find](https://scholar.google.com/scholar?q=N%2E+Djuric%2C+J%2E+Zhou%2C+R%2E+Morris%2C+M%2E+Grbovic%2C+V%2E+Radosavljevic%2C+and+N%2E+Bhamidipati%2E+2015%2E+Hate+speech+detection+with+comment+embeddings%2E+In+ICWWW%2C+pages+29%E2%80%9330%2E)
8. R. Dror, G. Baumer, S. Shlomov, and R. Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In ACL, Melbourne, Australia. [find](https://scholar.google.com/scholar?q=R%2E+Dror%2C+G%2E+Baumer%2C+S%2E+Shlomov%2C+and+R%2E+Reichart%2E+2018%2E+The+hitchhiker%E2%80%99s+guide+to+testing+statistical+significance+in+natural+language+processing%2E+In+ACL%2C+Melbourne%2C+Australia%2E)
9. M. ElSherief, V. Kulkarni, D. Nguyen, W. Y. Wang, and E. Belding. 2018. Hate lingo: A target-based linguistic analysis of hate speech in social media. arXiv preprint. [find](https://scholar.google.com/scholar?q=M%2E+ElSherief%2C+V%2E+Kulkarni%2C+D%2E+Nguyen%2C+W%2E+Y%2E+Wang%2C+and+E%2E+Belding%2E+2018%2E+Hate+lingo%3A+A+target%2Dbased+linguistic+analysis+of+hate+speech+in+social+media%2E+arXiv+preprint%2E)
10. B. Gambäck and U. K. Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In 1st Workshop on Abusive Language Online, pages 85–90, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=B%2E+Gamba%CC%88ck+and+U%2E+K%2E+Sikdar%2E+2017%2E+Using+convolutional+neural+networks+to+classify+hate%2Dspeech%2E+In+1st+Workshop+on+Abusive+Language+Online%2C+pages+85%E2%80%9390%2C+Vancouver%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/en/papers/gamback-2017-convolutional-neural-networks-classify-hate/)
11. L. Gao and R. Huang. 2017. Detecting online hate speech using context aware models. In RANLP, pages 260–266. [find](https://scholar.google.com/scholar?q=L%2E+Gao+and+R%2E+Huang%2E+2017%2E+Detecting+online+hate+speech+using+context+aware+models%2E+In+RANLP%2C+pages+260%E2%80%93266%2E)
12. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In ACL, pages 2786– 2791. [find](https://scholar.google.com/scholar?q=Kyle+Gorman+and+Steven+Bedrick%2E+2019%2E+We+need+to+talk+about+standard+splits%2E+In+ACL%2C+pages+2786%E2%80%93+2791%2E)
13. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%93+1780%2E)
14. H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran. 2017. Deceiving Google’s perspective api built for detecting toxic comments. arXiv preprint. [find](https://scholar.google.com/scholar?q=H%2E+Hosseini%2C+S%2E+Kannan%2C+B%2E+Zhang%2C+and+R%2E+Poovendran%2E+2017%2E+Deceiving+Google%E2%80%99s+perspective+api+built+for+detecting+toxic+comments%2E+arXiv+preprint%2E)
15. Y. Hua, C. Danescu-Niculescu-Mizil, D. Taraborelli, N. Thain, J. Sorensen, and L. Dixon. 2018. Wikiconv: A corpus of the complete conversational history of a large online collaborative community. arXiv preprint. [find](https://scholar.google.com/scholar?q=Y%2E+Hua%2C+C%2E+Danescu%2DNiculescu%2DMizil%2C+D%2E+Taraborelli%2C+N%2E+Thain%2C+J%2E+Sorensen%2C+and+L%2E+Dixon%2E+2018%2E+Wikiconv%3A+A+corpus+of+the+complete+conversational+history+of+a+large+online+collaborative+community%2E+arXi)
16. R. Kumar, A. K. Ojha, S. Malmasi, and M. Zampieri. 2018. Benchmarking aggression identification in social media. In TRAC, Santa Fe, USA. [find](https://scholar.google.com/scholar?q=R%2E+Kumar%2C+A%2E+K%2E+Ojha%2C+S%2E+Malmasi%2C+and+M%2E+Zampieri%2E+2018%2E+Benchmarking+aggression+identification+in+social+media%2E+In+TRAC%2C+Santa+Fe%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/kumar-2018-benchmarking-aggression-identification-social-media/)
17. S. Malmasi and M. Zampieri. 2017. Detecting hate speech in social media. In RANLP, pages 467–472. [find](https://scholar.google.com/scholar?q=S%2E+Malmasi+and+M%2E+Zampieri%2E+2017%2E+Detecting+hate+speech+in+social+media%2E+In+RANLP%2C+pages+467%E2%80%93472%2E)
18. T. Mikolov and G. Zweig. 2012. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234–239. IEEE. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov+and+G%2E+Zweig%2E+2012%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+2012+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2C+pages+234%E2%80%93239%2E+IEEE%2E)
19. H. Mubarak, K. Darwish, and W. Magdy. 2017. Abusive language detection on arabic social media. In 1st Abusive Language Workshop, pages 52–56, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=H%2E+Mubarak%2C+K%2E+Darwish%2C+and+W%2E+Magdy%2E+2017%2E+Abusive+language+detection+on+arabic+social+media%2E+In+1st+Abusive+Language+Workshop%2C+pages+52%E2%80%9356%2C+Vancouver%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/en/papers/mubarak-2017-abusive-language-detection-arabic-social/)
20. C. Nobata, J. Tetreault, A. Thomas, Y. Mehdad, and Y. Chang. 2016. Abusive language detection in online user content. In ICWWW, pages 145–153. [find](https://scholar.google.com/scholar?q=C%2E+Nobata%2C+J%2E+Tetreault%2C+A%2E+Thomas%2C+Y%2E+Mehdad%2C+and+Y%2E+Chang%2E+2016%2E+Abusive+language+detection+in+online+user+content%2E+In+ICWWW%2C+pages+145%E2%80%93153%2E)
21. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In ACL, pages 311–318. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2E+2002%2E+Bleu%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2C+pages+311%E2%80%93318%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
22. J. H. Park and P. Fung. 2017. One-step and two-step classification for abusive language detection on twitter. In 1st Workshop on Abusive Language Online, pages 41–45. [find](https://scholar.google.com/scholar?q=J%2E+H%2E+Park+and+P%2E+Fung%2E+2017%2E+One%2Dstep+and+two%2Dstep+classification+for+abusive+language+detection+on+twitter%2E+In+1st+Workshop+on+Abusive+Language+Online%2C+pages+41%E2%80%9345%2E) [our card](https://md.research.elamur.ai/free/en/papers/park-2017-one-step-two-step-classification/)
23. J. Pavlopoulos, P. Malakasiotis, and I. Androutsopoulos. 2017a. Deep learning for user comment moderation. In 1st Workshop on Abusive Language Online, pages 25–35. [find](https://scholar.google.com/scholar?q=J%2E+Pavlopoulos%2C+P%2E+Malakasiotis%2C+and+I%2E+Androutsopoulos%2E+2017a%2E+Deep+learning+for+user+comment+moderation%2E+In+1st+Workshop+on+Abusive+Language+Online%2C+pages+25%E2%80%9335%2E)
24. J. Pavlopoulos, P. Malakasiotis, and I. Androutsopoulos. 2017b. Deeper attention to abusive user content moderation. In EMNLP, pages 1125–1135, Copeng-hagen, Denmark. [find](https://scholar.google.com/scholar?q=J%2E+Pavlopoulos%2C+P%2E+Malakasiotis%2C+and+I%2E+Androutsopoulos%2E+2017b%2E+Deeper+attention+to+abusive+user+content+moderation%2E+In+EMNLP%2C+pages+1125%E2%80%931135%2C+Copeng%2Dhagen%2C+Denmark%2E)
25. L. Posch, A. Bleier, F. Flöck, and M. Strohmaier. 2018. Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint. [find](https://scholar.google.com/scholar?q=L%2E+Posch%2C+A%2E+Bleier%2C+F%2E+Flo%CC%88ck%2C+and+M%2E+Strohmaier%2E+2018%2E+Characterizing+the+global+crowd+workforce%3A+A+cross%2Dcountry+comparison+of+crowdworker+demographics%2E+arXiv+preprint%2E)
26. Y. Ren, Y. Zhang, M. Zhang, and D. Ji. 2016. Context-sensitive twitter sentiment classification using neural network. In 30th AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Y%2E+Ren%2C+Y%2E+Zhang%2C+M%2E+Zhang%2C+and+D%2E+Ji%2E+2016%2E+Context%2Dsensitive+twitter+sentiment+classification+using+neural+network%2E+In+30th+AAAI+Conference+on+Artificial+Intelligence%2E)
27. B. Ross, M. Rist, G. Carbonell, B. Cabrera, N. Kurowsky, and M. Wojatzki. 2016. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. In NLP4CMC, Bochum, Germany. [find](https://scholar.google.com/scholar?q=B%2E+Ross%2C+M%2E+Rist%2C+G%2E+Carbonell%2C+B%2E+Cabrera%2C+N%2E+Kurowsky%2C+and+M%2E+Wojatzki%2E+2016%2E+Measuring+the+reliability+of+hate+speech+annotations%3A+The+case+of+the+european+refugee+crisis%2E+In+NLP4CMC%2C+Bochum%2C+Germa)
28. M. D. Smucker, J. Allan, and B. Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In CIKM, pages 623– 632. ACM. [find](https://scholar.google.com/scholar?q=M%2E+D%2E+Smucker%2C+J%2E+Allan%2C+and+B%2E+Carterette%2E+2007%2E+A+comparison+of+statistical+significance+tests+for+information+retrieval+evaluation%2E+In+CIKM%2C+pages+623%E2%80%93+632%2E+ACM%2E)
29. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J.-Y. Nie, J. Gao, and B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint. [find](https://scholar.google.com/scholar?q=A%2E+Sordoni%2C+M%2E+Galley%2C+M%2E+Auli%2C+C%2E+Brockett%2C+Y%2E+Ji%2C+M%2E+Mitchell%2C+J%2E%2DY%2E+Nie%2C+J%2E+Gao%2C+and+B%2E+Dolan%2E+2015%2E+A+neural+network+approach+to+context%2Dsensitive+generation+of+conversational+responses%2E+arXiv+pre) [our card](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/)
30. Z. Waseem, T. Davidson, D. Warmsley, and I. Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In 1st Workshop on Abusive Langauge Online, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Z%2E+Waseem%2C+T%2E+Davidson%2C+D%2E+Warmsley%2C+and+I%2E+Weber%2E+2017%2E+Understanding+abuse%3A+A+typology+of+abusive+language+detection+subtasks%2E+In+1st+Workshop+on+Abusive+Langauge+Online%2C+Vancouver%2C+Canada%2E) [our card](https://md.research.elamur.ai/free/en/papers/waseem-2017-understanding-abuse-typology-abusive-language/)
31. Z. Waseem and D. Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In NAACL SRW, pages 88–93, San Diego, California. [find](https://scholar.google.com/scholar?q=Z%2E+Waseem+and+D%2E+Hovy%2E+2016%2E+Hateful+symbols+or+hateful+people%3F+predictive+features+for+hate+speech+detection+on+twitter%2E+In+NAACL+SRW%2C+pages+88%E2%80%9393%2C+San+Diego%2C+California%2E) [our card](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/)
32. M. Wiegand, M. Siegel, and J. Ruppenhofer. 2018. Overview of the germeval 2018 shared task on the identification of offensive language. In Proceedings of GermEval. [find](https://scholar.google.com/scholar?q=M%2E+Wiegand%2C+M%2E+Siegel%2C+and+J%2E+Ruppenhofer%2E+2018%2E+Overview+of+the+germeval+2018+shared+task+on+the+identification+of+offensive+language%2E+In+Proceedings+of+GermEval%2E)
33. E. Wulczyn, N. Thain, and L. Dixon. 2017. Ex machina: Personal attacks seen at scale. In ICWWW, pages 1391–1399. [find](https://scholar.google.com/scholar?q=E%2E+Wulczyn%2C+N%2E+Thain%2C+and+L%2E+Dixon%2E+2017%2E+Ex+machina%3A+Personal+attacks+seen+at+scale%2E+In+ICWWW%2C+pages+1391%E2%80%931399%2E)
34. M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In NAACL. [find](https://scholar.google.com/scholar?q=M%2E+Zampieri%2C+S%2E+Malmasi%2C+P%2E+Nakov%2C+S%2E+Rosenthal%2C+N%2E+Farra%2C+and+R%2E+Kumar%2E+2019a%2E+Predicting+the+Type+and+Target+of+Offensive+Posts+in+Social+Media%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/zampieri-2019-predicting-type-target-offensive-posts/)
35. M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar. 2019b. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In SemEval. [find](https://scholar.google.com/scholar?q=M%2E+Zampieri%2C+S%2E+Malmasi%2C+P%2E+Nakov%2C+S%2E+Rosenthal%2C+N%2E+Farra%2C+and+R%2E+Kumar%2E+2019b%2E+Semeval%2D2019+task+6%3A+Identifying+and+categorizing+offensive+language+in+social+media+%28offenseval%29%2E+In+SemEval%2E) [our card](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/)
36. Z. Zhang, D. Robinson, and J. Tepper. 2018. Detecting hate speech on twitter using a convolution-gru based deep neural network. In Lecture Notes in Computer Science. Springer Verlag. [find](https://scholar.google.com/scholar?q=Z%2E+Zhang%2C+D%2E+Robinson%2C+and+J%2E+Tepper%2E+2018%2E+Detecting+hate+speech+on+twitter+using+a+convolution%2Dgru+based+deep+neural+network%2E+In+Lecture+Notes+in+Computer+Science%2E+Springer+Verlag%2E)