---
title: "Utilizing Weak Supervision to Create S3D: A Sarcasm Annotated"
description: "The authors build two English Twitter resources. 78.84 recall. 32 references, cited by 3 papers here."
authors: ["Jordan Painter", "Helen Treharne", "Diptesh Kanojia"]
year: 2022
venue: "Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS)"
source: "https://aclanthology.org/2022.nlpcss-1.22/"
doi: "10.18653/v1/2022.nlpcss-1.22"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/painter-2022-utilizing-weak-supervision-create-s3d/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/painter-2022-utilizing-weak-supervision-create-s3d/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / Utilizing Weak Supervision to Create S3D: A Sarcasm Annotated Dataset

# Utilizing Weak Supervision to Create S3D: A Sarcasm Annotated Dataset

**Authors:** Jordan Painter, Helen Treharne and [Diptesh Kanojia](https://md.research.elamur.ai/free/en/authors/diptesh-kanojia/) · **Year:** 2022 · **Venue:** [Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS)](https://md.research.elamur.ai/free/en/venues/nlpcss/) ([NLP+CSS 2022](https://md.research.elamur.ai/free/en/venues/nlpcss/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.nlpcss-1.22/) · [PDF](https://aclanthology.org/2022.nlpcss-1.22.pdf) · [DOI 10.18653/v1/2022.nlpcss-1.22](https://doi.org/10.18653/v1/2022.nlpcss-1.22) · [OpenAlex](https://openalex.org/W4385567074)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/painter-2022-utilizing-weak-supervision-create-s3d/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Sarcasm detection on social media is still largely unsolved because sarcastic text often says the opposite of what it means, which hurts tasks like sentiment analysis. The authors build two English Twitter resources. SAD is a small, manually checked gold set collected through the #sarcasm hashtag and paired with ordinary tweets from the same users. S3D is a large silver set produced by automatic labelling. They merge four benchmark corpora (SARC, Ptacek, SemEval2018, Riloff) with SAD into a "Combined" corpus of about one million items. On that corpus they compare classical classifiers with Transformer models. A BERTweet model fine-tuned on Combined reaches an F1 of 78.29. A majority-vote ensemble of three strong fine-tuned models then labels 100,000 newly scraped tweets. Adding S3D to the training data raises F1 slightly, to 78.87. The work shows that weak supervision can grow or refresh sarcasm corpora, which matters because tweet-ID datasets lose content as tweets are deleted. Code, data and models are publicly released.

## Key points
- SAD has 2,340 tweets, of which 1,170 carried #sarcasm. For each username there is one sarcastic and one non-sarcastic tweet. Two annotators labelled the data and a third, blind annotator resolved disputes. Cohen's Kappa was 0.83, and the manual labels matched the hashtag self-labels in 98% of cases.
- The Combined corpus (SARC, Ptacek, SemEval2018, Riloff, SAD) holds 1,022,546 Reddit and Twitter texts, split roughly evenly between sarcastic and non-sarcastic.
- BERTweet fine-tuned on Combined scores F1 78.29, and RoBERTa large on Combined scores 78.01.
- The S3D labeller is a majority vote over three models:
  - BERTweet trained on SARC,
  - BERTweet trained on Combined,
  - RoBERTa large trained on Combined.
- S3D contains 100,000 tweets posted 7–9 September 2022, and close to 40% were labelled sarcastic. Fine-tuning BERTweet on Combined plus S3D gives F1 78.87 (precision 78.84, recall 78.89).
- Tweet-ID datasets decay over time: only 710 of Riloff's original 3,000 tweets could still be retrieved. The released datasets therefore contain only tweet IDs and labels, with user data stripped out.

## Interesting ideas
- Sampling one sarcastic and one non-sarcastic tweet from each user keeps author identity and style from becoming a shortcut, so the classifier has to rely on the text itself
- Noisy self-applied tags such as #sarcasm were checked against careful manual labels, and the 98% agreement measures how trustworthy the cheap signal is
- Silver labels came from a majority vote of models that differ in architecture and training data, not from a single model
- The authors manually reviewed silver-labelled positives and found that many could be read as sincere or sarcastic depending on context nobody could see
- Automatically labelled data is proposed as a way to top up gold datasets that shrink as their sources disappear

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): When building irony test sets for crate prag, pair ironic and literal sentences from the same source or author. A detector then cannot win on style cues alone.
- **I2** (learn): Before trusting any cheap label source as silver data, hand-check a sample of it. Record the agreement rate as a gate in the evaluation report.
- **I3** (learn): This matches the witness-parser setup. Keep the witnesses deliberately diverse (different parsers, different training treebanks) so that their errors are less correlated before accepting their agreed output.
- **I4** (frames): The irony component should be allowed to output "ambiguous without context" rather than a forced yes/no. It should also be able to explain which missing world fact, such as how serious the event is, would settle the reading.
- **I5** (learn): When a gold resource cannot be fully reproduced, refill it with witness-agreed silver examples. Keep the gold and silver parts clearly tagged so evaluation runs only on gold.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Surrey](https://md.research.elamur.ai/free/en/universities/university-of-surrey/) — University, Guildford, United Kingdom · 7 works · 566 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works

## Cite

Cite as: Jordan Painter, Helen Treharne and Diptesh Kanojia. 2022. Utilizing Weak Supervision to Create S3D: A Sarcasm Annotated Dataset. Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{painter2022utilizing,
  title = {{Utilizing Weak Supervision to Create S3D: A Sarcasm Annotated Dataset}},
  author = {Jordan Painter, Helen Treharne and Diptesh Kanojia},
  booktitle = {Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS)},
  year = {2022},
  doi = {10.18653/v1/2022.nlpcss-1.22},
  url = {https://doi.org/10.18653/v1/2022.nlpcss-1.22}
}
```

- [← previous UoR-NCL at SemEval-2022 Task 6: Using ensemble loss with BERT for intended sarcasm detection](https://md.research.elamur.ai/free/en/papers/brefo-2022-uor-ncl-semeval-2022-task/)
- [next → AlexU-AL at SemEval-2022 Task 6: Detecting Sarcasm in Arabic Text Using Deep Learning Techniques](https://md.research.elamur.ai/free/en/papers/lotfy-2022-alexu-al-semeval-2022-task/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

32 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. [ARBERT & MARBERT: deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). CoRR, abs/2101.01785. [↗](https://arxiv.org/abs/2101.01785)
2. Ibrahim Abu Farha and Walid Magdy. 2021. [Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection](https://md.research.elamur.ai/free/en/papers/farha-2021-benchmarking-transformer-based-language-models/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 21–31, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
3. Francesco Barbieri, José Camacho-Collados, Leonardo Neves, and Luis Espinosa Anke. 2020. [Tweeteval: Unified benchmark and comparative evaluation for tweet classification](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/). CoRR, abs/2010.12421. [↗](https://arxiv.org/abs/2010.12421)
4. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. [Modelling sarcasm in Twitter, a novel approach](https://md.research.elamur.ai/free/en/papers/barbieri-2014-modelling-sarcasm-twitter-novel-approach/). In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58, Baltimore, Maryland. Association for Computational Linguistics.
5. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. [Towards multimodal sarcasm detection (an _Obviously_ perfect paper)](https://md.research.elamur.ai/free/en/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1455)
6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
7. David R Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2):215–232. [↗](https://doi.org/10.1111/j.2517-6161.1958.tb00292.x)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
10. Megan L. Dress, Roger J. Kreuz, Kristen E. Link, and Gina M. Caucci. 2008. Regional variation in the use of sarcasm. Journal of Language and Social Psychology, 27:71 – 85. [↗](https://doi.org/10.1177/0261927x07309512)
11. Mohsen Farhadloo and Erik Rolland. 2016. Fundamentals of sentiment analysis and its applications. In Sentiment analysis and ontology engineering, pages 1–24. Springer. [↗](https://doi.org/10.1007/978-3-319-30319-2_1)
12. Aniruddha Ghosh and Tony Veale. 2016. [Fracking sarcasm using neural network](https://md.research.elamur.ai/free/en/papers/ghosh-2016-fracking-sarcasm-neural-network/). In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169, San Diego, California. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w16-0425)
13. Priya Goel, Rachna Jain, Anand Nayyar, Shruti Singhal, and Muskan Srivastava. 2022. Sarcasm detection using deep learning and ensemble learning. Multimedia Tools and Applications. [↗](https://doi.org/10.1007/s11042-022-12930-z)
14. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. [Automatic sarcasm detection: A survey](https://md.research.elamur.ai/free/en/papers/joshi-2016-automatic-sarcasm-detection-survey/). ACM Comput. Surv., 50(5). [↗](https://doi.org/10.48550/arxiv.1602.03426)
15. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. [A large self-annotated corpus for sarcasm](https://md.research.elamur.ai/free/en/papers/khodak-2018-large-self-annotated-corpus-sarcasm/).
16. Hyafil Laurent and Ronald L Rivest. 1976. Constructing optimal binary decision trees is np-complete. Information processing letters, 5(1):15–17. [↗](https://doi.org/10.1016/0020-0190(76)90095-8)
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
18. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 4068–4074. [find](https://scholar.google.com/scholar?q=Dehong+Ma%2C+Sujian+Li%2C+Xiaodong+Zhang%2C+and+Houfeng+Wang%2E+2017%2E+Interactive+attention+networks+for+aspect%2Dlevel+sentiment+classification%2E+In+Proceedings+of+the+Twenty%2DSixth+International+Joint+Conferenc)
19. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). [↗](https://doi.org/10.48550/arxiv.1301.3781)
20. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. [BERTweet: A pre-trained language model for English tweets](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14, Online. Association for Computational Linguistics.
21. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2(1–2):1–135. [↗](https://doi.org/10.1561/9781601981516)
22. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830. [find](https://scholar.google.com/scholar?q=Fabian+Pedregosa%2C+Ga%C3%ABl+Varoquaux%2C+Alexandre+Gramfort%2C+Vincent+Michel%2C+Bertrand+Thirion%2C+Olivier+Grisel%2C+Mathieu+Blondel%2C+Peter+Prettenhofer%2C+Ron+Weiss%2C+Vincent+Dubourg%2C+et+al%2E+2011%2E+Scikit%2Dlearn%3A+Mach)
23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. [↗](https://doi.org/10.3115/v1/d14-1162)
24. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2019. [A transformer-based approach to irony and sarcasm detection](https://md.research.elamur.ai/free/en/papers/potamias-2019-transformer-based-approach-irony-sarcasm/). CoRR, abs/1911.10401. [↗](https://arxiv.org/abs/1911.10401)
25. Tomás Ptácek, Ivan Habernal, and Jun Hong. 2014. [Sarcasm detection on czech and english twitter](https://md.research.elamur.ai/free/en/papers/ptacek-2014-sarcasm-detection-czech-english-twitter/). In COL-ING.
26. Anupama Ray, Shubham Mishra, Apoorva Nunna, and Pushpak Bhattacharyya. 2022. [A multimodal corpus for emotion recognition in sarcasm](https://md.research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/). In Proceedings of the Language Resources and Evaluation Conference, pages 6992–7003, Marseille, France. European Language Resources Association.
27. Antonio Reyes and Paolo Rosso. 2011. [Mining subjective knowledge from customer reviews: A specific case of irony detection](https://md.research.elamur.ai/free/en/papers/reyes-2011-mining-subjective-knowledge-customer-reviews/). In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pages 118–124, Portland, Oregon. Association for Computational Linguistics.
28. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. [Sarcasm as contrast between a positive sentiment and negative situation](https://md.research.elamur.ai/free/en/papers/riloff-2013-sarcasm-as-contrast-between-positive/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d13-1066)
29. Taha Shangipour ataei, Soroush Javdan, and Behrouz Minaei-Bidgoli. 2020. [Applying transformers and aspect-based sentiment analysis approaches on sarcasm detection](https://md.research.elamur.ai/free/en/papers/ataei-2020-applying-transformers-aspect-based-sentiment/). In Proceedings of the Second Workshop on Figurative Language Processing, pages 67– 71, Online. Association for Computational Linguistics.
30. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. [SemEval-2018 task 3: Irony detection in English tweets](https://md.research.elamur.ai/free/en/papers/hee-2018-semeval-2018-task-3-irony/). In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s18-1005)
31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
32. Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731–5780. [↗](https://doi.org/10.1007/s10462-022-10144-1)

*Updated 24 October 2026*