---
title: "SemEval-2021 Task 5 — Pavlopoulos 2021"
description: "The paper describes a SemEval-2021 shared task. Instead of labelling a whole comment as toxic, systems had to locate the character spans that make it…"
authors: ["John Pavlopoulos", "Jeffrey Sorensen", "Léo Laugier", "Ion Androutsopoulos"]
year: 2021
venue: "Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)"
source: "https://aclanthology.org/2021.semeval-1.6/"
doi: "10.18653/v1/2021.semeval-1.6"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pavlopoulos-2021-semeval-2021-task-5-toxic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pavlopoulos-2021-semeval-2021-task-5-toxic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / SemEval-2021 Task 5: Toxic Spans Detection

# SemEval-2021 Task 5: Toxic Spans Detection

**Authors:** [John Pavlopoulos](https://md.research.elamur.ai/free/en/authors/john-pavlopoulos/), [Jeffrey Sorensen](https://md.research.elamur.ai/free/en/authors/jeffrey-sorensen/), Léo Laugier, [Ion Androutsopoulos](https://md.research.elamur.ai/free/en/authors/ion-androutsopoulos/) · **Institutions:** [Google](https://md.research.elamur.ai/free/en/companies/google/) · [Athens University of Economics and Business](https://md.research.elamur.ai/free/en/universities/athens-university-of-economics-and-business/) · [Stockholm University](https://md.research.elamur.ai/free/en/universities/stockholm-university/) · [Institut Polytechnique de Paris](https://md.research.elamur.ai/free/en/universities/institut-polytechnique-de-paris/) · [Télécom Paris](https://md.research.elamur.ai/free/en/universities/telecom-paris/) · **Year:** 2021 · **Venue:** Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.semeval-1.6/) · [PDF](https://aclanthology.org/2021.semeval-1.6.pdf) · [DOI 10.18653/v1/2021.semeval-1.6](https://doi.org/10.18653/v1/2021.semeval-1.6) · [OpenAlex](https://openalex.org/W3184890107)

**Citations:** 80 citations (OpenAlex, 2026-10-14) · [92](https://md.research.elamur.ai/free/en/papers/pavlopoulos-2021-semeval-2021-task-5-toxic/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper describes a SemEval-2021 shared task. Instead of labelling a whole comment as toxic, systems had to locate the character spans that make it toxic, so that moderators can see at a glance what is objectionable. The organisers took about 30k comments from Civil Comments that crowd raters had judged toxic. On the Appen platform, crowd workers re-annotated part of them by highlighting offending text. Per-rater highlights were merged by majority vote at the character level, producing the ToxicSpans dataset. Submissions were scored with character-based F1 on a held-out test set. Participants could treat the problem as supervised sequence labelling or as rationale extraction from post-level toxicity classifiers. Around 500 people formed 92 teams, and 36 teams submitted system descriptions. Fine-tuned pre-trained Transformers, especially in ensembles, performed best. Rationale-based methods lagged behind but need no span-level labels. The paper identifies three open problems: long context-dependent spans, posts without any toxic span, and annotation noise.

## Key points
- **Dataset:** ToxicSpans is released under CC0 at github.com/ipavlopoulos/toxic_spans.
  - Splits are 690 trial, 7,939 train and 2,000 test posts.
  - Test posts are shorter and have fewer and shorter spans on average.
- **Annotation process:**
  - Up to seven raters annotated each post; test posts had three to seven.
  - A character is gold-toxic if most raters marked it.
  - A gold span is a maximal run of contiguous toxic characters.
- **Agreement:** Mean pairwise Cohen's Kappa, computed over character offsets, was 0.61 on 35 posts. Larger samples of up to 1,000 posts stayed above 0.55.
- **Evaluation schedule:** The evaluation phase ran from 10 to 31 January 2021. The daily submission cap fell from 10 to 5 to 1 per team per week, to discourage overfitting the test set.
- **Systems:**
  - The winning team, HITSZ-HLT, ensembled token-labelling models with a span-extraction component. The token-labelling models were BERT with a CRF layer, one also with an LSTM layer.
  - Rationale-extraction entries paired LSTM, LR, SVM or BERT classifiers with Input Erasure, attention, LIME or SHAP.

## Interesting ideas
- **Character-level gold labels from majority vote.** Merging annotators by character-wise majority, then forming spans from contiguous runs, avoids tokenizer mismatches. This could suit any span-annotation project, including Ukrainian data with rich morphology.
- **Agreement measured on character offsets.** Treating each character as a binary classification instance makes Kappa computable for free-form highlighting. Many span tasks lack a natural unit, so this offers a usable agreement measure.
- **Shrinking submission quotas.** Allowing many uploads early and only one per day at the end keeps engagement high while limiting test-set tuning. This is a cheap safeguard for any leaderboard-based evaluation.
- **Rationale extraction as a label-free baseline.** Explanations from a post-level classifier (attention, LIME, SHAP, erasure) can propose spans without span annotation. This is attractive for low-resource languages where only document-level labels exist.
- **Category priming during annotation.** Raters chose among five toxicity types, even though the categories were later collapsed into one. This primed raters to think about what kind of toxicity they saw. The same trick could sharpen annotator attention in other subjective labelling tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pavlopoulos2021semeval,
  title = {{SemEval-2021 Task 5: Toxic Spans Detection}},
  author = {Pavlopoulos, John and Sorensen, Jeffrey and Laugier, Léo and Androutsopoulos, Ion},
  booktitle = {Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)},
  year = {2021},
  doi = {10.18653/v1/2021.semeval-1.6},
  url = {https://doi.org/10.18653/v1/2021.semeval-1.6}
}
```

- [← previous Stance Detection in COVID-19 Tweets](https://md.research.elamur.ai/free/en/papers/glandt-2021-stance-detection-covid-19-tweets/)
- [next → Findings of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion](https://md.research.elamur.ai/free/en/papers/chakravarthi-2021-findings-shared-task-hope-speech/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

41 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In NAACL Demonstrations, pages 54–59. [find](https://scholar.google.com/scholar?q=Alan+Akbik%2C+Tanja+Bergmann%2C+Duncan+Blythe%2C+Kashif+Rasul%2C+Stefan+Schweter%2C+and+Roland+Vollgraf%2E+2019%2E+Flair%3A+An+easy%2Dto%2Duse+framework+for+state%2Dof%2Dthe%2Dart+nlp%2E+In+NAACL+Demonstrations%2C+pages+54%E2%80%9359%2E) [our card](https://md.research.elamur.ai/free/en/papers/akbik-2019-flair-easy-use-framework-state/)
2. Archit Bansal, Abhay Kaushik, and Ashutosh Modi. 2021. IITK@Detox at SemEval-2021 Task 5: Semi-Birol Kuyumcu, Selman Delil, and Cüneyt aksakallı. 2021. Sefamerve arge at SemEval-2021 Task 5: Toxic span detection using segmentation based 1-d convolutional neural network model. In SemEval. supervised… [find](https://scholar.google.com/scholar?q=Archit+Bansal%2C+Abhay+Kaushik%2C+and+Ashutosh+Modi%2E+2021%2E+IITK%40Detox+at+SemEval%2D2021+Task+5%3A+Semi%2DBirol+Kuyumcu%2C+Selman+Delil%2C+and+Cu%CC%88neyt+aksakall%C4%B1%2E+2021%2E+Sefamerve+arge+at+SemEval%2D2021+Task+5%3A+Toxic+sp)
3. Abdessamad Benlahbib, Hamza Alami, and Ahmed Alami. 2021. LISAC FSDM USMBA at SemEval 2021 Task 5: Tackling toxic spans detection challenge with supervised spanBERT-based model and unsupervised LIME-based model. In SemEval. [find](https://scholar.google.com/scholar?q=Abdessamad+Benlahbib%2C+Hamza+Alami%2C+and+Ahmed+Alami%2E+2021%2E+LISAC+FSDM+USMBA+at+SemEval+2021+Task+5%3A+Tackling+toxic+spans+detection+challenge+with+supervised+spanBERT%2Dbased+model+and+unsupervised+LIME%2Db)
4. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016%2E+Understanding+neural+networks+through+representation+erasure%2E+arXiv+preprint+arXiv%3A1612%2E08220%2E+%E2%80%94+%5BarXiv%3A1612%2E08220%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E08220%29)
5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
6. Aja Bogdanoff. 2017. Saying goodbye to civil comments. Accessed: 2021-04-15. [find](https://scholar.google.com/scholar?q=Aja+Bogdanoff%2E+2017%2E+Saying+goodbye+to+civil+comments%2E+Accessed%3A+2021%2D04%2D15%2E)
7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+TACL%2C+5%3A135%E2%80%93146%2E)
8. Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874. [find](https://scholar.google.com/scholar?q=Scott+Lundberg+and+Su%2DIn+Lee%2E+2017%2E+A+unified+approach+to+interpreting+model+predictions%2E+arXiv+preprint+arXiv%3A1705%2E07874%2E) [our card](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/)
9. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019a. Nuanced metrics for measuring unintended bias with real data for text classification. In WWW, pages 491–500, San Francisco, USA. [find](https://scholar.google.com/scholar?q=Daniel+Borkan%2C+Lucas+Dixon%2C+Jeffrey+Sorensen%2C+Nithum+Thain%2C+and+Lucy+Vasserman%2E+2019a%2E+Nuanced+metrics+for+measuring+unintended+bias+with+real+data+for+text+classification%2E+In+WWW%2C+pages+491%E2%80%93500%2C+San+)
10. Son T. Luu and Ngan Nguyen. 2021. UIT-ISE-NLP at SemEval-2021 Task 5: Toxic span detection with BiLSTM - CRF and toxic BERT comment classification. In SemEval. [find](https://scholar.google.com/scholar?q=Son+T%2E+Luu+and+Ngan+Nguyen%2E+2021%2E+UIT%2DISE%2DNLP+at+SemEval%2D2021+Task+5%3A+Toxic+span+detection+with+BiLSTM+%2D+CRF+and+toxic+BERT+comment+classification%2E+In+SemEval%2E)
11. Daniel Borkan, Jeffrey Sorensen, and Lucy Vasserman. 2019b. Exploring the role of human raters in creating nlp datasets. Accessed: 2021-04-15. [find](https://scholar.google.com/scholar?q=Daniel+Borkan%2C+Jeffrey+Sorensen%2C+and+Lucy+Vasserman%2E+2019b%2E+Exploring+the+role+of+human+raters+in+creating+nlp+datasets%2E+Accessed%3A+2021%2D04%2D15%2E)
12. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In EMNLP-IJCNLP, pages 5640– 5650. [find](https://scholar.google.com/scholar?q=Giovanni+Da+San+Martino%2C+Seunghak+Yu%2C+Alberto+Barro%CC%81n%2DCedeno%2C+Rostislav+Petrov%2C+and+Preslav+Nakov%2E+2019%2E+Fine%2Dgrained+analysis+of+propaganda+in+news+article%2E+In+EMNLP%2DIJCNLP%2C+pages+5640%E2%80%93+5650%2E)
13. Ben Burtenshaw and Mike Kestemont. 2021. UAntwerp at SemEval-2021 Task 5: Spans are spans, stacking a binary word level approach to toxic span detection. In SemEval. [find](https://scholar.google.com/scholar?q=Ben+Burtenshaw+and+Mike+Kestemont%2E+2021%2E+UAntwerp+at+SemEval%2D2021+Task+5%3A+Spans+are+spans%2C+stacking+a+binary+word+level+approach+to+toxic+span+detection%2E+In+SemEval%2E)
14. Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020. Towards transparent and explainable attention models. In ACL, pages 4206–4216. [find](https://scholar.google.com/scholar?q=Akash+Kumar+Mohankumar%2C+Preksha+Nema%2C+Sharan+Narasimhan%2C+Mitesh+M+Khapra%2C+Balaji+Vasan+Srinivasan%2C+and+Balaraman+Ravindran%2E+2020%2E+Towards+transparent+and+explainable+attention+models%2E+In+ACL%2C+pages+42)
15. Maggie Cech. 2021. macech at SemEval-2021 Task 5: Toxic spans detection. In SemEval. [find](https://scholar.google.com/scholar?q=Maggie+Cech%2E+2021%2E+macech+at+SemEval%2D2021+Task+5%3A+Toxic+spans+detection%2E+In+SemEval%2E)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2C+pages+4171%E2%80%934186%2E)
17. Viet Anh Nguyen, Tam Nguyen, Huy Dao Quang, and Quang Huu Pham. 2021. S-NLP at semeval-2021 task 5: Toxic spans detection. In SemEval. [find](https://scholar.google.com/scholar?q=Viet+Anh+Nguyen%2C+Tam+Nguyen%2C+Huy+Dao+Quang%2C+and+Quang+Huu+Pham%2E+2021%2E+S%2DNLP+at+semeval%2D2021+task+5%3A+Toxic+spans+detection%2E+In+SemEval%2E)
18. Huiyang Ding and David Jurgens. 2021. HamiltonD-inggg at SemEval-2021 Task 5: Investigating toxic span detection using RoBERTa pre-training. In SemEval. [find](https://scholar.google.com/scholar?q=Huiyang+Ding+and+David+Jurgens%2E+2021%2E+HamiltonD%2Dinggg+at+SemEval%2D2021+Task+5%3A+Investigating+toxic+span+detection+using+RoBERTa+pre%2Dtraining%2E+In+SemEval%2E)
19. Marco Palomino, Dawid Grad, and James Bedwell. 2021. An ensemble approach to identify toxicity in text. In SemEval. [find](https://scholar.google.com/scholar?q=Marco+Palomino%2C+Dawid+Grad%2C+and+James+Bedwell%2E+2021%2E+An+ensemble+approach+to+identify+toxicity+in+text%2E+In+SemEval%2E)
20. Kline Finley. 2016. Want to save the comments from trolls? do it yourself. Accessed: 2021-04-15. [find](https://scholar.google.com/scholar?q=Kline+Finley%2E+2016%2E+Want+to+save+the+comments+from+trolls%3F+do+it+yourself%2E+Accessed%3A+2021%2D04%2D15%2E)
21. John Pavlopoulos, Prodromos Malakasiotis, and Ion Androutsopoulos. 2017. Deeper attention to abusive user content moderation. In EMNLP, pages 1125– 1135, Copenghagen, Denmark. [find](https://scholar.google.com/scholar?q=John+Pavlopoulos%2C+Prodromos+Malakasiotis%2C+and+Ion+Androutsopoulos%2E+2017%2E+Deeper+attention+to+abusive+user+content+moderation%2E+In+EMNLP%2C+pages+1125%E2%80%93+1135%2C+Copenghagen%2C+Denmark%2E)
22. Sreyan Ghosh and Sonal Kumar. 2021. Cisco at SemEval-2021 Task 5: What’s toxic?: Leveraging transformers for multiple toxic span extraction from online comments. In SemEval. [find](https://scholar.google.com/scholar?q=Sreyan+Ghosh+and+Sonal+Kumar%2E+2021%2E+Cisco+at+SemEval%2D2021+Task+5%3A+What%E2%80%99s+toxic%3F%3A+Leveraging+transformers+for+multiple+toxic+span+extraction+from+online+comments%2E+In+SemEval%2E)
23. Kamil Pluciński and Hanna Klimczak. 2021. GHOST at SemEval-2021 Task 5: Is explanation all you need? In SemEval. [find](https://scholar.google.com/scholar?q=Kamil+Plucin%CC%81ski+and+Hanna+Klimczak%2E+2021%2E+GHOST+at+SemEval%2D2021+Task+5%3A+Is+explanation+all+you+need%3F+In+SemEval%2E)
24. Phu Gia Hoang and Luan Thanh Nguyen. 2021. UIT-E10dot3 at SemEval 2021 Task 5: Toxic spans detection with roberta and spacy’s library base systems. In SemEval. [find](https://scholar.google.com/scholar?q=Phu+Gia+Hoang+and+Luan+Thanh+Nguyen%2E+2021%2E+UIT%2DE10dot3+at+SemEval+2021+Task+5%3A+Toxic+spans+detection+with+roberta+and+spacy%E2%80%99s+library+base+systems%2E+In+SemEval%2E)
25. Tharindu Ranasinghe, Diptanu Sarkar, Marcos Zampieri, and Alexander Ororbia. 2021. WLV-RIT at SemEval-2021 Task 5: A neural transformer framework for detecting toxic spans. In SemEval. [find](https://scholar.google.com/scholar?q=Tharindu+Ranasinghe%2C+Diptanu+Sarkar%2C+Marcos+Zampieri%2C+and+Alexander+Ororbia%2E+2021%2E+WLV%2DRIT+at+SemEval%2D2021+Task+5%3A+A+neural+transformer+framework+for+detecting+toxic+spans%2E+In+SemEval%2E)
26. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
27. Marco T. Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why Should I Trust You?” Explaining the predictions of any classifier. In SIGKDD, pages 1135–1144, San Francisco, USA. [find](https://scholar.google.com/scholar?q=Marco+T%2E+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016%2E+%E2%80%9CWhy+Should+I+Trust+You%3F%E2%80%9D+Explaining+the+predictions+of+any+classifier%2E+In+SIGKDD%2C+pages+1135%E2%80%931144%2C+San+Francisco%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/)
28. Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. In ACL, pages 537–546. [find](https://scholar.google.com/scholar?q=Minghao+Hu%2C+Yuxing+Peng%2C+Zhen+Huang%2C+Dongsheng+Li%2C+and+Yiwei+Lv%2E+2019%2E+Open%2Ddomain+targeted+sentiment+analysis+via+span%2Dbased+extraction+and+classification%2E+In+ACL%2C+pages+537%E2%80%93546%2E)
29. Jonathan Rusert. 2021. NLP UIOWA at Semeval-2021 Task 5: Transferring toxic sets to tag toxic spans. In SemEval. [find](https://scholar.google.com/scholar?q=Jonathan+Rusert%2E+2021%2E+NLP+UIOWA+at+Semeval%2D2021+Task+5%3A+Transferring+toxic+sets+to+tag+toxic+spans%2E+In+SemEval%2E)
30. Rafel Palliser Sans and Albert Rial Farràs. 2021. HLE-UPC at SemEval-2021 Task 5: Multi-Depth Distil-BERT for toxic spans detection. In SemEval. [find](https://scholar.google.com/scholar?q=Rafel+Palliser+Sans+and+Albert+Rial+Farra%CC%80s%2E+2021%2E+HLE%2DUPC+at+SemEval%2D2021+Task+5%3A+Multi%2DDepth+Distil%2DBERT+for+toxic+spans+detection%2E+In+SemEval%2E)
31. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain. [find](https://scholar.google.com/scholar?q=Anna+Schmidt+and+Michael+Wiegand%2E+2017%2E+A+survey+on+hate+speech+detection+using+natural+language+processing%2E+In+Workshop+on+Natural+Language+Processing+for+Social+Media%2C+pages+1%E2%80%9310%2C+Valencia%2C+Spain%2E) [our card](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/)
32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+ACL%2E)
33. Tom De Smedt, Pierre Voué, Sylvia Jaki, Melina Röttcher, and Guy De Pauw. 2020. Profanity & offensive words (POW). Textgain. [find](https://scholar.google.com/scholar?q=Tom+De+Smedt%2C+Pierre+Voue%CC%81%2C+Sylvia+Jaki%2C+Melina+Ro%CC%88ttcher%2C+and+Guy+De+Pauw%2E+2020%2E+Profanity+%26+offensive+words+%28POW%29%2E+Textgain%2E)
34. Thakur Ashutosh Suman and Abhinav Jain. 2021. AS-tarTwice at SemEval-2021 Task 5: Toxic span detection using RoBERTa-CRF, domain specific pre-training and self-training. In SemEval. [find](https://scholar.google.com/scholar?q=Thakur+Ashutosh+Suman+and+Abhinav+Jain%2E+2021%2E+AS%2DtarTwice+at+SemEval%2D2021+Task+5%3A+Toxic+span+detection+using+RoBERTa%2DCRF%2C+domain+specific+pre%2Dtraining+and+self%2Dtraining%2E+In+SemEval%2E)
35. Charles Sutton and Andrew McCallum. 2006. An Introduction to Conditional Random Fields for relational learning. Introduction to statistical relational learning, 2:93–128. [find](https://scholar.google.com/scholar?q=Charles+Sutton+and+Andrew+McCallum%2E+2006%2E+An+Introduction+to+Conditional+Random+Fields+for+relational+learning%2E+Introduction+to+statistical+relational+learning%2C+2%3A93%E2%80%93128%2E)
36. Amos Tversky. 1977. Features of similarity. Psychological review, 84(4):327. [find](https://scholar.google.com/scholar?q=Amos+Tversky%2E+1977%2E+Features+of+similarity%2E+Psychological+review%2C+84%284%29%3A327%2E)
37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, volume 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2C+volume+30%2E)
38. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In WWW, pages 1391–1399, Perth, Australia. [find](https://scholar.google.com/scholar?q=Ellery+Wulczyn%2C+Nithum+Thain%2C+and+Lucas+Dixon%2E+2017%2E+Ex+machina%3A+Personal+attacks+seen+at+scale%2E+In+WWW%2C+pages+1391%E2%80%931399%2C+Perth%2C+Australia%2E)
39. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In SemEval. [find](https://scholar.google.com/scholar?q=Marcos+Zampieri%2C+Shervin+Malmasi%2C+Preslav+Nakov%2C+Sara+Rosenthal%2C+Noura+Farra%2C+and+Ritesh+Kumar%2E+2019%2E+Semeval%2D2019+task+6%3A+Identifying+and+categorizing+offensive+language+in+social+media+%28offenseval%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/)
40. Hongjie Fan Zhen Wang and Junfei Liu. 2021. MedAI at SemEval-2021 Task 5: Start-to-end tagging framework for toxic spans detection. In SemEval. [find](https://scholar.google.com/scholar?q=Hongjie+Fan+Zhen+Wang+and+Junfei+Liu%2E+2021%2E+MedAI+at+SemEval%2D2021+Task+5%3A+Start%2Dto%2Dend+tagging+framework+for+toxic+spans+detection%2E+In+SemEval%2E)
41. Qinglin Zhu, Zijie Lin, Yice Zhang, Jingyi Sun, Xiang Li, Qihui Lin, and Ruifeng Xu. 2021. HITSZ-HLT at SemEval-2021 Task 5: Span-based ensemble model with toxic lexicon. In SemEval. [find](https://scholar.google.com/scholar?q=Qinglin+Zhu%2C+Zijie+Lin%2C+Yice+Zhang%2C+Jingyi+Sun%2C+Xiang+Li%2C+Qihui+Lin%2C+and+Ruifeng+Xu%2E+2021%2E+HITSZ%2DHLT+at+SemEval%2D2021+Task+5%3A+Span%2Dbased+ensemble+model+with+toxic+lexicon%2E+In+SemEval%2E)