---
title: "NULI at SemEval-2019 Task 6 — Liu 2019"
description: "This paper describes team NULI's system for SemEval-2019 Task 6."
authors: ["Ping Liu", "Wen Li", "Liang Zou"]
year: 2019
venue: "Proceedings of the 13th International Workshop on Semantic Evaluation"
source: "https://aclanthology.org/S19-2011/"
doi: "10.18653/v1/s19-2011"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2019-nuli-semeval-2019-task-6/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2019-nuli-semeval-2019-task-6/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / NULI at SemEval-2019 Task 6: Transfer Learning for Offensive Language Detection using Bidirectional Transformers

# NULI at SemEval-2019 Task 6: Transfer Learning for Offensive Language Detection using Bidirectional Transformers

**Authors:** Ping Liu, Wen Li, Liang Zou · **Year:** 2019 · **Venue:** Proceedings of the 13th International Workshop on Semantic Evaluation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/S19-2011/) · [PDF](https://aclanthology.org/S19-2011.pdf) · [DOI 10.18653/v1/s19-2011](https://doi.org/10.18653/v1/s19-2011) · [OpenAlex](https://openalex.org/W2953553271)

**Citations:** 171 citations (openalex-snapshot · 2026-10-14) · [81](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This paper describes team NULI's system for SemEval-2019 Task 6. The task is to detect and categorize offensive tweets in the OLID dataset across three hierarchical sub-tasks: whether a post is offensive (A), whether an offense is targeted or untargeted (B), and whether the target is an individual, a group or something else (C). The authors identify three main difficulties: little training data, noisy social-media language, and class imbalance that differs between the training and test splits.

Their preprocessing focuses on tokens that standard vocabularies lack. Emoji are converted into English phrases, hashtags are split into separate words, and user mentions and URLs are normalized. They compare three classifiers:
- a Logistic Regression model using bag-of-words, word2vec and Hatebase features;
- an LSTM;
- a fine-tuned pre-trained BERT.

BERT performed best and was used for the official runs. It placed 1st in sub-task A, 4th in B and 18th in C. The results show that transfer learning from a pre-trained model is a strong starting point when labeled data is scarce.

## Key points
- **Data imbalance:** OLID training data contains 8840 NOT vs 4400 OFF tweets, 3876 TIN vs 524 UNT, and 2407 IND / 1074 GRP / 395 OTH. Class proportions shift between splits; for example, OTH makes up about 10.1% of training data but 16.4% of test data.
- **Preprocessing tools:** emoji were mapped to text with the `carpedm20/emoji` library, and hashtags were segmented with `python-wordsegment` (e.g., `#LunaticLeft` becomes "Lunatic Left"). Text was also lowercased, "URL" was replaced by "http", and runs of `@USER` were capped at three.
- **Official test results (macro-F1):**

  | Sub-task | Model | Macro-F1 | Accuracy | Comparison |
  |---|---|---|---|---|
  | A | BERT-3 | 0.8286 | 0.8628 | Bagging over 50 Logistic Regression models: 0.7558; single linear model: 0.7501 |
  | B | BERT-3 | 0.7159 | — | — |
  | C | BERT-2 | 0.5598 | — | — |

- **LSTM baseline:** 140-dimensional embeddings, 64 hidden units, dropout of 0.5, and a maximum sequence length of 140.
- **Why B and C scored lower:** the authors attribute this to smaller training sets and more skewed classes. The rare OTH class in sub-task C was especially hard.

## Interesting ideas
- **Turn emoji and hashtags into plain words.** Converting them before feeding text to a pre-trained model keeps their meaning, which would otherwise be lost as unknown tokens. This is cheap and applies to any social-media text, including Ukrainian.
- **Match placeholders to the pre-trained vocabulary.** Swapping a placeholder like "URL" for a token the model already knows ("http") is a small trick. It is worth checking whenever a dataset uses special markers that a pre-trained vocabulary may not cover.
- **Report constant-prediction baselines.** Scores for systems that always predict one class show how misleading accuracy can be under imbalance, since a high-accuracy majority guess still gets poor macro-F1. This is a useful sanity check for any skewed classification benchmark.
- **Save checkpoints from each early epoch.** BERT fine-tuning converged within one to two epochs on this small dataset, so the authors evaluated models after epochs 1, 2 and 3 instead of training long. With little data, cheap checkpoint comparison like this can matter more than heavy tuning.
- **Check for distribution shift between splits.** Comparing class proportions in training vs test data helps explain weak results on minority classes. It also motivates balancing strategies such as weighted loss, which the authors mention as an option.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/)

## Cited by

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 74 not yet.

- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://md.research.elamur.ai/free/en/papers/caselli-2021-hatebert-retraining-bert-abusive-language/)
- [A Corpus of Turkish Offensive Language on Social Media](https://md.research.elamur.ai/free/en/papers/coltekin-2020-corpus-turkish-offensive-language-social/)
- [I Feel Offended, Don't Be Abusive! Implicit/Explicit Messages in Offensive and Abusive Language](https://md.research.elamur.ai/free/en/papers/caselli-2020-i-feel-offended-don-t/)
- [Detecting Sarcasm in Conversation Context Using Transformer-Based Models](https://md.research.elamur.ai/free/en/papers/avvaru-2020-detecting-sarcasm-conversation-context-transformer/)
- [Offensive language detection in Hebrew: can other languages help?](https://md.research.elamur.ai/free/en/papers/lingbulk-2022-litvak-offensive-language-detection-hebrew-can/)
- [From Linguistics to Practice: a Case Study of Offensive Language Taxonomy in Hebrew](https://md.research.elamur.ai/free/en/papers/aclbulk-2024-liebeskind-linguistics-practice-case-study-offensive/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2019nuli,
  title = {{NULI at SemEval-2019 Task 6: Transfer Learning for Offensive Language Detection using Bidirectional Transformers}},
  author = {Liu, Ping and Li, Wen and Zou, Liang},
  booktitle = {Proceedings of the 13th International Workshop on Semantic Evaluation},
  year = {2019},
  doi = {10.18653/v1/s19-2011},
  url = {https://doi.org/10.18653/v1/s19-2011}
}
```

- [← previous Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://md.research.elamur.ai/free/en/papers/he-2019-unlearn-dataset-bias-natural-language/)
- [next → BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jun-Ming Xu, Kwang-Sung Jun, Xiaojin Zhu, and Amy Bellmore. 2012. Learning from bullying traces in social media. In Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 656–666. Association for Computati… [find](https://scholar.google.com/scholar?q=Jun%2DMing+Xu%2C+Kwang%2DSung+Jun%2C+Xiaojin+Zhu%2C+and+Amy+Bellmore%2E+2012%2E+Learning+from+bullying+traces+in+social+media%2E+In+Proceedings+of+the+2012+conference+of+the+North+American+chapter+of+the+association+) [our card](https://md.research.elamur.ai/free/en/papers/xu-2012-learning-bullying-traces-social-media/)
2. Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pages 1217–1230. ACM. [find](https://scholar.google.com/scholar?q=Justin+Cheng%2C+Michael+Bernstein%2C+Cristian+Danescu%2DNiculescu%2DMizil%2C+and+Jure+Leskovec%2E+2017%2E+Anyone+can+become+a+troll%3A+Causes+of+trolling+behavior+in+online+discussions%2E+In+Proceedings+of+the+2017+ACM)
3. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Marcos+Zampieri%2C+Shervin+Malmasi%2C+Preslav+Nakov%2C+Sara+Rosenthal%2C+Noura+Farra%2C+and+Ritesh+Kumar%2E+2019a%2E+Predicting+the+Type+and+Target+of+Offensive+Posts+in+Social+Media%2E+In+Proceedings+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/zampieri-2019-predicting-type-target-offensive-posts/)
4. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of ICWSM. [find](https://scholar.google.com/scholar?q=Thomas+Davidson%2C+Dana+Warmsley%2C+Michael+Macy%2C+and+Ingmar+Weber%2E+2017%2E+Automated+Hate+Speech+Detection+and+the+Problem+of+Offensive+Language%2E+In+Proceedings+of+ICWSM%2E) [our card](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)
5. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval). [find](https://scholar.google.com/scholar?q=Marcos+Zampieri%2C+Shervin+Malmasi%2C+Preslav+Nakov%2C+Sara+Rosenthal%2C+Noura+Farra%2C+and+Ritesh+Kumar%2E+2019b%2E+SemEval%2D2019+Task+6%3A+Identifying+and+Categorizing+Offensive+Language+in+Social+Media+%28OffensEval%29) [our card](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
7. Darja Fišer, Tomaž Erjavec, and Nikola Ljubešić. 2017. Legal Framework, Dataset and Annotation Schema for Socially Unacceptable On-line Discourse Practices in Slovene. In Proceedings of the Workshop Workshop on Abusive Language Online (ALW), Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Darja+Fis%CC%8Cer%2C+Tomaz%CC%8C+Erjavec%2C+and+Nikola+Ljubes%CC%8Cic%CC%81%2E+2017%2E+Legal+Framework%2C+Dataset+and+Annotation+Schema+for+Socially+Unacceptable+On%2Dline+Discourse+Practices+in+Slovene%2E+In+Proceedings+of+the+Worksh)
8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
9. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting Tweets Against Blacks. In Twenty-Seventh AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Irene+Kwok+and+Yuzhou+Wang%2E+2013%2E+Locate+the+hate%3A+Detecting+Tweets+Against+Blacks%2E+In+Twenty%2DSeventh+AAAI+Conference+on+Artificial+Intelligence%2E)
10. Ping Liu, Joshua Guberman, Libby Hemphill, and Aron Culotta. 2018. Forecasting the presence and intensity of hostility on instagram using linguistic and social features. In Twelfth International AAAI Conference on Web and Social Media. [find](https://scholar.google.com/scholar?q=Ping+Liu%2C+Joshua+Guberman%2C+Libby+Hemphill%2C+and+Aron+Culotta%2E+2018%2E+Forecasting+the+presence+and+intensity+of+hostility+on+instagram+using+linguistic+and+social+features%2E+In+Twelfth+International+AAAI+)
11. Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection Using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, pages 1–10, Valencia, Spain. [find](https://scholar.google.com/scholar?q=Anna+Schmidt+and+Michael+Wiegand%2E+2017%2E+A+Survey+on+Hate+Speech+Detection+Using+Natural+Language+Processing%2E+In+Proceedings+of+the+Fifth+International+Workshop+on+Natural+Language+Processing+for+Socia) [our card](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/)
12. Huei-Po Su, Chen-Jie Huang, Hao-Tsung Chang, and Chuan-Jie Lin. 2017. Rephrasing Profanity in Chinese Text. In Proceedings of the Workshop Workshop on Abusive Language Online (ALW), Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Huei%2DPo+Su%2C+Chen%2DJie+Huang%2C+Hao%2DTsung+Chang%2C+and+Chuan%2DJie+Lin%2E+2017%2E+Rephrasing+Profanity+in+Chinese+Text%2E+In+Proceedings+of+the+Workshop+Workshop+on+Abusive+Language+Online+%28ALW%29%2C+Vancouver%2C+Canada%2E)
13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
14. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In Proceedings of the First Workshop on Abusive Langauge Online. [find](https://scholar.google.com/scholar?q=Zeerak+Waseem%2C+Thomas+Davidson%2C+Dana+Warmsley%2C+and+Ingmar+Weber%2E+2017%2E+Understanding+Abuse%3A+A+Typology+of+Abusive+Language+Detection+Subtasks%2E+In+Proceedings+of+the+First+Workshop+on+Abusive+Langauge+) [our card](https://md.research.elamur.ai/free/en/papers/waseem-2017-understanding-abuse-typology-abusive-language/)