---
title: "Putting Sarcasm Detection into Context — Abercrombie 2016"
description: "The paper argues that typical sarcasm detection experiments are unrealistic in two ways."
authors: ["Gavin Abercrombie", "Dirk Hovy"]
year: 2016
venue: "Proceedings of the ACL 2016 Student Research Workshop"
source: "https://aclanthology.org/P16-3016/"
doi: "10.18653/v1/p16-3016"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/abercrombie-2016-putting-sarcasm-detection-into-context/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/abercrombie-2016-putting-sarcasm-detection-into-context/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2016](https://md.research.elamur.ai/free/papers/2016/page/3/) / Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations

# Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations

**Authors:** Gavin Abercrombie, [Dirk Hovy](https://md.research.elamur.ai/free/authors/dirk-hovy/) · **Year:** 2016 · **Venue:** Proceedings of the ACL 2016 Student Research Workshop · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P16-3016/) · [PDF](https://aclanthology.org/P16-3016.pdf) · [DOI 10.18653/v1/p16-3016](https://doi.org/10.18653/v1/p16-3016) · [OpenAlex](https://openalex.org/W2513973860)

**Citations:** 63 citations (openalex-snapshot · 2026-10-14) · [22](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper argues that typical sarcasm detection experiments are unrealistic in two ways. They use evenly split datasets even though sarcasm is uncommon, and they rely on hashtags like #sarcasm as labels, which are noisy and capture only some kinds of sarcasm. To test how much these choices matter, the authors built a hand-annotated corpus of two-tweet Twitter conversations in which roughly one in five replies is sarcastic. They also built a hashtag-labelled corpus of the same size for comparison. Human raters and an L2-regularised logistic regression model using the Bamman and Smith (2015) feature classes were evaluated under five context conditions. Humans consistently outperformed the classifier and gained more from context, though both did worst when given all information at once. Classifier scores were higher on the hashtag-based data and fell on the imbalanced splits, most sharply for F1. The authors conclude that reported sarcasm detection results depend heavily on labelling method and class prevalence. They suggest that a true gold standard would need labels from people who share the conversational common ground.

## Key points
- **Corpus construction:** Conversations were drawn from a 2013 collection of 64M tweets, which yielded 650,212 English two-line conversations. Manual annotation identified 448 sarcastic replies, which were combined with 1,792 non-sarcastic ones for 2,240 conversations in total. A hashtag-based corpus was built with the same proportions, with #sarcasm removed from the text.
- **Human study:** 60 native English speakers each rated 10 of 300 conversations, with two raters per item. The five conditions were tweet only, + author, + audience, + environment, and all combined. Inter-rater agreement was low, and the highest agreement came in the tweet-only condition.
- **Human rater scores:** The MACE "most competent" rater reached F1 0.547 and AUC 0.731. The best rater per pair reached F1 0.641 and AUC 0.817.
- **Classifier setup and results:** The model was evaluated with five-fold cross-validation and reported with F1 and AUC. It beat random chance in every condition but stayed well below humans. Its best AUC was 0.630 with tweet + environment features, compared with 0.802 for humans in that condition, and its lowest was 0.615 with tweet + author features.
- **Effects of imbalance and labelling:** Imbalanced data caused large F1 drops, and AUC also declined in most configurations. The hashtag corpus produced generally higher scores, which the authors attribute to more homogeneous data and to bias toward particular user types.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

22 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 11 not yet.

- [Detecting Sarcasm in Conversation Context Using Transformer-Based Models](https://md.research.elamur.ai/free/papers/avvaru-2020-detecting-sarcasm-conversation-context-transformer/)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://md.research.elamur.ai/free/papers/castro-2019-towards-multimodal-sarcasm-detection-obviously/)
- [From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset](https://md.research.elamur.ai/free/papers/farha-2020-arabic-sentiment-analysis-sarcasm-detection/)
- [Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic](https://md.research.elamur.ai/free/papers/farha-2021-overview-wanlp-2021-shared-task/)
- [Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection](https://md.research.elamur.ai/free/papers/farha-2022-sarcasm-detection-is-way-too/)
- [SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/papers/farha-2022-semeval-2022-task-6-isarcasmeval/)
- [The Importance of Modeling Social Factors of Language: Theory and Practice](https://md.research.elamur.ai/free/papers/hovy-2021-importance-modeling-social-factors-language/)
- [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://md.research.elamur.ai/free/papers/jing-2023-multi-source-semantic-graph-based/)
- [Automatic Sarcasm Detection: A Survey](https://md.research.elamur.ai/free/papers/joshi-2016-automatic-sarcasm-detection-survey/)
- [iSarcasm: A Dataset of Intended Sarcasm](https://md.research.elamur.ai/free/papers/oprea-2020-isarcasm-dataset-intended-sarcasm/)
- [A deep-learning framework to detect sarcasm targets](https://md.research.elamur.ai/free/papers/patro-2019-deep-learning-framework-detect-sarcasm/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{abercrombie2016putting,
  title = {{Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations}},
  author = {Abercrombie, Gavin and Hovy, Dirk},
  booktitle = {Proceedings of the ACL 2016 Student Research Workshop},
  year = {2016},
  doi = {10.18653/v1/p16-3016},
  url = {https://doi.org/10.18653/v1/p16-3016}
}
```

- [← previous Predicting and Using a Pragmatic Component of Lexical Aspect of Simple Past Verbal Tenses for Improving English-to-French Machine Translation](https://md.research.elamur.ai/free/papers/loaiciga-2016-predicting-pragmatic-component-lexical-aspect/)
- [next → Reasoning about Pragmatics with Neural Listeners and Speakers](https://md.research.elamur.ai/free/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/papers/2016/page/3/)

## References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Ninth International AAAI Conference on Web and Social Media. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Noah+A+Smith%2E+2015%2E+Contextualized+sarcasm+detection+on+twitter%2E+In+Ninth+International+AAAI+Conference+on+Web+and+Social+Media%2E)
2. Diana Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Diana+Maynard+and+Mark+A+Greenwood%2E+2014%2E+Who+cares+about+sarcastic+tweets%3F+investigating+the+impact+of+sarcasm+on+sentiment+analysis%2E+In+Proceedings+of+LREC%2E) [our card](https://md.research.elamur.ai/free/papers/maynard-2014-who-cares-about-sarcastic-tweets/)
3. Paula Carvalho, Luı́s Sarmento, Mário J Silva, and Eugénio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-). In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. ACM. [find](https://scholar.google.com/scholar?q=Paula+Carvalho%2C+Lu%C4%B1%CC%81s+Sarmento%2C+Ma%CC%81rio+J+Silva%2C+and+Euge%CC%81nio+De+Oliveira%2E+2009%2E+Clues+for+detecting+irony+in+user%2Dgenerated+contents%3A+oh%2E%2E%2E%21%21+it%E2%80%99s+so+easy%3B%2D%29%2E+In+Proceedings+of+the+1st+international+C)
4. D. M. W. Powers. 2015. What the F-measure doesn’t measure: Features, Flaws, Fallacies and Fixes. ArXiv e-prints, March. [↗](https://doi.org/10.13140/rg.2.2.23168.79361)
5. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Oren+Tsur%2C+and+Ari+Rappoport%2E+2010%2E+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+twitter+and+amazon%2E+In+Proceedings+of+the+Fourteenth+Conference+on+Computational+Natural+Langu) [our card](https://md.research.elamur.ai/free/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
6. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. [find](https://scholar.google.com/scholar?q=Ashwin+Rajadesingan%2C+Reza+Zafarani%2C+and+Huan+Liu%2E+2015%2E+Sarcasm+detection+on+twitter%3A+A+behavioral+modeling+approach%2E+In+Proceedings+of+the+Eighth+ACM+International+Conference+on+Web+Search+and+Data+M)
7. Tom Fawcett. 2004. Roc graphs: Notes and practical considerations for researchers. Machine learning, 31(1):1–38. [find](https://scholar.google.com/scholar?q=Tom+Fawcett%2E+2004%2E+Roc+graphs%3A+Notes+and+practical+considerations+for+researchers%2E+Machine+learning%2C+31%281%29%3A1%E2%80%9338%2E)
8. Reid Swanson, Stephanie M Lukin, Luke Eisenberg, Thomas Corcoran, and Marilyn A Walker. 2014. Getting reliable annotations for sarcasm in online dialogues. In LREC, pages 4250–4257. [find](https://scholar.google.com/scholar?q=Reid+Swanson%2C+Stephanie+M+Lukin%2C+Luke+Eisenberg%2C+Thomas+Corcoran%2C+and+Marilyn+A+Walker%2E+2014%2E+Getting+reliable+annotations+for+sarcasm+in+online+dialogues%2E+In+LREC%2C+pages+4250%E2%80%934257%2E) [our card](https://md.research.elamur.ai/free/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/)
9. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In LREC, pages 392–398. [find](https://scholar.google.com/scholar?q=Elena+Filatova%2E+2012%2E+Irony+and+sarcasm%3A+Corpus+generation+and+analysis+using+crowdsourcing%2E+In+LREC%2C+pages+392%E2%80%93398%2E) [our card](https://md.research.elamur.ai/free/papers/filatova-2012-irony-sarcasm-corpus-generation-analysis/)
10. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm-a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In ICWSM. [find](https://scholar.google.com/scholar?q=Oren+Tsur%2C+Dmitry+Davidov%2C+and+Ari+Rappoport%2E+2010%2E+Icwsm%2Da+great+catchy+name%3A+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+online+product+reviews%2E+In+ICWSM%2E)
11. Raymond Gibbs and Herbert Colston. 2007. The future of irony studies. Irony in Language and Thought, pages 339–360. [find](https://scholar.google.com/scholar?q=Raymond+Gibbs+and+Herbert+Colston%2E+2007%2E+The+future+of+irony+studies%2E+Irony+in+Language+and+Thought%2C+pages+339%E2%80%93360%2E)
12. Akira Utsumi. 2000. Verbal irony as implicit display of ironic environment: Distinguishing ironic utterances from nonirony. Journal of Pragmatics, 32(12):1777–1806. [↗](https://doi.org/10.1016/s0378-2166(99)00116-2)
13. Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of Experimental Psychology: General, 115(1):3. [↗](https://doi.org/10.1037//0096-3445.115.1.3)
14. Byron C Wallace, Laura Kertz Do Kook Choe, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). [find](https://scholar.google.com/scholar?q=Byron+C+Wallace%2C+Laura+Kertz+Do+Kook+Choe%2C+and+Eugene+Charniak%2E+2014%2E+Humans+require+context+to+infer+ironic+intent+%28so+computers+probably+do%2C+too%29%2E)
15. Raymond W Gibbs. 2000. Irony in talk among friends. Metaphor and symbol, 15(1-2):5–27. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 512–516. [↗](https://doi.org/10.1080/10926488.2000.9678862)