---
title: "Reactive Supervision — Shmueli 2020"
description: "Sarcasm detection models need large labeled corpora."
authors: ["Boaz Shmueli", "Lun-Wei Ku", "Soumya Ray"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.201/"
doi: "10.18653/v1/2020.emnlp-main.201"
arxiv: "2009.13080"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/shmueli-2020-reactive-supervision-new-method-collecting/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/shmueli-2020-reactive-supervision-new-method-collecting/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2020](https://md.research.elamur.ai/free/papers/2020/page/4/) / Reactive Supervision: A New Method for Collecting Sarcasm Data

# Reactive Supervision: A New Method for Collecting Sarcasm Data

**Authors:** Boaz Shmueli, Lun-Wei Ku, Soumya Ray · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.201/) · [PDF](https://aclanthology.org/2020.emnlp-main.201.pdf) · [DOI 10.18653/v1/2020.emnlp-main.201](https://doi.org/10.18653/v1/2020.emnlp-main.201) · [arXiv 2009.13080](https://arxiv.org/abs/2009.13080) · [OpenAlex](https://openalex.org/W3089146887)

**Citations:** 23 citations (openalex-snapshot · 2026-10-14) · [13](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Sarcasm detection models need large labeled corpora. Existing ways of gathering such data each have drawbacks. Hashtag-based distant supervision is noisy and captures only the author's intent. Manual annotation and manual collection are slow, costly and capture a single perspective. The authors propose reactive supervision, which mines Twitter conversations for cue replies containing the phrase "being sarcastic." The grammatical person of the cue (I / you / s/he) is mapped to a regular expression over the thread's author sequence. This locates the sarcastic tweet unambiguously. First-person cues mark intended sarcasm, while second- and third-person cues mark perceived sarcasm. The resulting SPIRS dataset also includes the oblivious reply (a reader who missed the sarcasm) and the eliciting tweet. Manual checks show labeling accuracy of about 97–98.5%. Baselines with CNN, BiLSTM and BERT cover three tasks: sarcasm detection, context-aware detection and a new intended-vs-perceived perspective task. The resource matters because it is the first to provide both sarcasm perspectives and oblivious texts. The method could also be extended to emotion and sentiment data.

## Key points
- **Collection pipeline:** The pipeline ran Fetch, Classify, Traverse and Match over 48 days in Oct–Nov 2019. It gathered 65K cue tweets, and 77% were discarded as unclassifiable.
- **SPIRS contents:** The dataset holds 15,000 English sarcastic tweets, split into 10,300 intended and 4,700 perceived (3,000 second-person, 1,700 third-person). It also includes 10,648 oblivious tweets, 9,156 eliciting tweets and 15,000 random negatives.
- **Collection rate:** The method gathered 312 tweets/day, compared with 171/day for #sarcasm distant supervision, an 82% improvement.
- **Sarcasm detection:** BERT reaches 70.3% accuracy on SPIRS. The Ptáček dataset scores higher (86.6%), which the authors attribute to locale confounding between its classes.
- **Context and perspective:** A three-BiLSTM context model achieves 74.7% accuracy. Dropping oblivious tweets cuts accuracy to 71.4% and MCC from 0.398 to 0.275. On perspective classification, BERT reaches 68.2% accuracy (MCC 0.366).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 10 not yet.

- [Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection](https://md.research.elamur.ai/free/papers/farha-2022-sarcasm-detection-is-way-too/)
- [SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/papers/farha-2022-semeval-2022-task-6-isarcasmeval/)
- [A Survey in Automatic Irony Processing: Linguistic, Cognitive, and Multi-X Perspectives](https://md.research.elamur.ai/free/papers/zeng-2022-survey-automatic-irony-processing-linguistic/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shmueli2020reactive,
  title = {{Reactive Supervision: A New Method for Collecting Sarcasm Data}},
  author = {Shmueli, Boaz and Ku, Lun-Wei and Ray, Soumya},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.201},
  eprint = {2009.13080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.201}
}
```

- [← previous R^3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge](https://md.research.elamur.ai/free/papers/chakrabarty-2020-r-textasciicircum3-reverse-retrieve-rank/)
- [next → Real-Valued Logics for Typological Universals: Framework and Application](https://md.research.elamur.ai/free/papers/donicke-2020-real-valued-logics-typological-universals/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/papers/2020/page/4/)

## References

16 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Bamman and Noah A Smith. 2015. Contextualized Sarcasm Detection on Twitter. In Ninth International AAAI Conference on Web and Social Media. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Noah+A+Smith%2E+2015%2E+Contextualized+Sarcasm+Detection+on+Twitter%2E+In+Ninth+International+AAAI+Conference+on+Web+and+Social+Media%2E)
2. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised Recognition of Sarcastic Sentences in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Stroudsburg, PA, USA. Association for Computational Linguistics. Event-place: Uppsala, Sweden. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Oren+Tsur%2C+and+Ari+Rappoport%2E+2010%2E+Semi%2Dsupervised+Recognition+of+Sarcastic+Sentences+in+Twitter+and+Amazon%2E+In+Proceedings+of+the+Fourteenth+Conference+on+Computational+Natural+Langu) [our card](https://md.research.elamur.ai/free/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
4. Elena Filatova. 2012. Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 392–398, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Elena+Filatova%2E+2012%2E+Irony+and+Sarcasm%3A+Corpus+Generation+and+Analysis+Using+Crowdsourcing%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages) [our card](https://md.research.elamur.ai/free/papers/filatova-2012-irony-sarcasm-corpus-generation-analysis/)
5. Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of Experimental Psychology: General, 115(1):3. [↗](https://doi.org/10.1037//0096-3445.115.1.3)
6. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying Sarcasm in Twitter: A Closer Look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 581–586, Stroudsburg, PA, USA. Association for Computational Linguistics. Event-place: Portland, Oregon. [find](https://scholar.google.com/scholar?q=Roberto+Gonza%CC%81lez%2DIba%CC%81n%CC%83ez%2C+Smaranda+Muresan%2C+and+Nina+Wacholder%2E+2011%2E+Identifying+Sarcasm+in+Twitter%3A+A+Closer+Look%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Li) [our card](https://md.research.elamur.ai/free/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/)
7. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst%2E+1992%2E+Automatic+acquisition+of+hyponyms+from+large+text+corpora%2E+In+COLING+1992+Volume+2%3A+The+15th+International+Conference+on+Computational+Linguistics%2E)
8. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016. How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 95–99, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aditya+Joshi%2C+Pushpak+Bhattacharyya%2C+Mark+Carman%2C+Jaya+Saraswati%2C+and+Rajita+Shukla%2E+2016%2E+How+Do+Cultural+Differences+Impact+the+Quality+of+Sarcasm+Annotation%3F%3A+A+Case+Study+of+Indian+Annotators+and+) [our card](https://md.research.elamur.ai/free/papers/joshi-2016-how-do-cultural-differences-impact/)
9. Christine Liebrecht, Florian Kunneman, and Antal van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets #not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christine+Liebrecht%2C+Florian+Kunneman%2C+and+Antal+van+den+Bosch%2E+2013%2E+The+perfect+solution+for+detecting+sarcasm+in+tweets+%23not%2E+In+Proceedings+of+the+4th+Workshop+on+Computational+Approaches+to+Subje) [our card](https://md.research.elamur.ai/free/papers/liebrecht-2013-perfect-solution-detecting-sarcasm-tweets/)
10. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854–2859, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1275) [our card](https://md.research.elamur.ai/free/papers/oprea-2019-exploring-author-context-detecting-intended/)
11. Silviu Oprea and Walid Magdy. 2020. iSarcasm: A Dataset of Intended Sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Silviu+Oprea+and+Walid+Magdy%2E+2020%2E+iSarcasm%3A+A+Dataset+of+Intended+Sarcasm%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E+Association+for+Computational+L) [our card](https://md.research.elamur.ai/free/papers/oprea-2020-isarcasm-dataset-intended-sarcasm/)
12. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. [↗](https://doi.org/10.3115/v1/d14-1162)
13. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm Detection on Czech and English Twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Pta%CC%81c%CC%8Cek%2C+Ivan+Habernal%2C+and+Jun+Hong%2E+2014%2E+Sarcasm+Detection+on+Czech+and+English+Twitter%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Conference+on+Computational+Linguistics%3A+Tech) [our card](https://md.research.elamur.ai/free/papers/ptacek-2014-sarcasm-detection-czech-english-twitter/)
14. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as Contrast between a Positive Sentiment and Negative Situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ellen+Riloff%2C+Ashequl+Qadir%2C+Prafulla+Surve%2C+Lalindra+De+Silva%2C+Nathan+Gilbert%2C+and+Ruihong+Huang%2E+2013%2E+Sarcasm+as+Contrast+between+a+Positive+Sentiment+and+Negative+Situation%2E+In+Proceedings+of+the+) [our card](https://md.research.elamur.ai/free/papers/prag-2013-riloff-sarcasm-as-contrast-between-positive/)
15. Reid Swanson, Stephanie Lukin, Luke Eisenberg, Thomas Corcoran, and Marilyn Walker. 2014. Getting reliable annotations for sarcasm in online dialogues. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4250–4257, Reykjavik, Iceland. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Reid+Swanson%2C+Stephanie+Lukin%2C+Luke+Eisenberg%2C+Thomas+Corcoran%2C+and+Marilyn+Walker%2E+2014%2E+Getting+reliable+annotations+for+sarcasm+in+online+dialogues%2E+In+Proceedings+of+the+Ninth+International+Confer) [our card](https://md.research.elamur.ai/free/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/)
16. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with Sarcasm by Reading In-Between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1010–1020, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Anh+Tuan+Luu%2C+Siu+Cheung+Hui%2C+and+Jian+Su%2E+2018%2E+Reasoning+with+Sarcasm+by+Reading+In%2DBetween%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum) [our card](https://md.research.elamur.ai/free/papers/tay-2018-reasoning-sarcasm-reading-between/)