--- title: "Identifying Sarcasm in Twitter — González-Ibáñez 2011" description: "This paper looks at automatic sarcasm detection on Twitter." authors: ["Roberto González-Ibáñez", "Smaranda Muresan", "Nina Wacholder"] year: 2011 venue: "Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies" source: "https://aclanthology.org/P11-2102/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Identifying Sarcasm in Twitter: A Closer Look # Identifying Sarcasm in Twitter: A Closer Look **Authors:** Roberto González-Ibáñez, [Smaranda Muresan](https://md.research.elamur.ai/free/en/authors/smaranda-muresan/), Nina Wacholder · **Year:** 2011 · **Venue:** Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P11-2102/) · [PDF](https://aclanthology.org/P11-2102.pdf) · [OpenAlex](https://openalex.org/W2250489604) **Citations:** 621 citations (openalex-snapshot · 2026-10-14) · [121](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This paper looks at automatic sarcasm detection on Twitter. Progress on the task has been held back by a shortage of reliably labelled, naturally occurring examples. To fix this, the authors build a corpus where the tweet authors mark sarcasm themselves with #sarcasm or #sarcastic. These tweets are compared with tweets tagged with plain positive or negative emotion hashtags. The data is cleaned automatically and then by hand, keeping only tweets where the hashtag ends the message. That leaves 900 English tweets per class. The authors then test two kinds of cues with SMO and logistic regression classifiers. Lexical cues are unigrams, LIWC categories, WordNet Affect, interjections and punctuation. Pragmatic cues are emoticons and whether the tweet replies to another user (ToUser). The systems are also compared with human judges. Telling sarcasm apart from plain sentiment turns out to be hard: the best three-way accuracy is about 57%, and the human judges do no better. The study matters for two reasons. It shows that labels from the authors make a more reliable gold standard than labels from outside coders. It also suggests that surface word features are not enough and that context and shared knowledge are needed. ## Key points - Corpus: 2,700 tweets, 900 each for sarcastic, positive and negative, labelled by the authors' own hashtags. Retweets, URLs, non-English tweets and tweets where the hashtag was not at the end were removed. - Best classifier accuracies with 5-fold cross-validation: S-P-N 57.22% (SMO, unigrams); sarcastic vs. non-sarcastic 65.44%; S-P and S-N close to 70%; positive vs. negative 75.89%. - In the χ² ranking, the three pragmatic features (ToUser, smiley, frown) are among the top 10 most useful features. Sarcastic tweets have positive-emotion words like positive tweets and more negation like negative tweets. - Human judges did poorly. In the three-way task their mean accuracy was 62.59%, falling to 43.33% when only unanimous labels were counted, and agreement was moderate (Fleiss' κ 0.4788). In the two-way task the classifiers (up to 68.33%) beat the humans. - On a 100-tweet subset that all contain emoticons, humans reached 73% and SMO 71%. Judges said missing context and missing world knowledge were the main obstacles. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) ## Cited by Cited by 121 papers of our library cite it (of the 140884 whose bibliographies we read); 58 of them have a page here, 63 not yet. - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [Sarcasm as Contrast between a Positive Sentiment and Negative Situation](https://md.research.elamur.ai/free/en/papers/prag-2013-riloff-sarcasm-as-contrast-between-positive/) - [SemEval-2014 Task 9: Sentiment Analysis in Twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2014-semeval-2014-task-9-sentiment/) - [SemEval-2015 Task 10: Sentiment Analysis in Twitter](https://md.research.elamur.ai/free/en/papers/rosenthal-2015-semeval-2015-task-10-sentiment/) - [SemEval-2018 Task 3: Irony Detection in English Tweets](https://md.research.elamur.ai/free/en/papers/hee-2018-semeval-2018-task-3-irony/) - [CARER: Contextualized Affect Representations for Emotion Recognition](https://md.research.elamur.ai/free/en/papers/saravia-2018-carer-contextualized-affect-representations-emotion/) - [Harnessing Context Incongruity for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/joshi-2015-harnessing-context-incongruity-sarcasm-detection/) - [That's So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets](https://md.research.elamur.ai/free/en/papers/wang-2015-that-s-so-annoying-lexical/) - [Fracking Sarcasm using Neural Network](https://md.research.elamur.ai/free/en/papers/ghosh-2016-fracking-sarcasm-neural-network/) - [A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/poria-2016-deeper-look-into-sarcastic-tweets/) - [A Transformer-based approach to Irony and Sarcasm detection](https://md.research.elamur.ai/free/en/papers/potamias-2019-transformer-based-approach-irony-sarcasm/) - [Detecting Sarcasm in Multimodal Social Platforms](https://md.research.elamur.ai/free/en/papers/schifanella-2016-detecting-sarcasm-multimodal-social-platforms/) - [A Large Self-Annotated Corpus for Sarcasm](https://md.research.elamur.ai/free/en/papers/khodak-2018-large-self-annotated-corpus-sarcasm/) - [Reasoning with Sarcasm by Reading In-Between](https://md.research.elamur.ai/free/en/papers/tay-2018-reasoning-sarcasm-reading-between/) - [Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing](https://md.research.elamur.ai/free/en/papers/filatova-2012-irony-sarcasm-corpus-generation-analysis/) - [Modelling Sarcasm in Twitter, a Novel Approach](https://md.research.elamur.ai/free/en/papers/barbieri-2014-modelling-sarcasm-twitter-novel-approach/) - [Automatic Sarcasm Detection: A Survey](https://md.research.elamur.ai/free/en/papers/joshi-2016-automatic-sarcasm-detection-survey/) - [Are Word Embedding-based Features Useful for Sarcasm Detection?](https://md.research.elamur.ai/free/en/papers/joshi-2016-are-word-embedding-based-features/) - [Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association](https://md.research.elamur.ai/free/en/papers/xu-2020-reasoning-multimodal-sarcastic-tweets-modeling/) - [Tweet Sarcasm Detection Using Deep Neural Network](https://md.research.elamur.ai/free/en/papers/zhang-2016-tweet-sarcasm-detection-deep-neural/) - [Magnets for Sarcasm: Making Sarcasm Detection Timely, Contextual and Very Personal](https://md.research.elamur.ai/free/en/papers/ghosh-2017-magnets-sarcasm-making-sarcasm-detection/) - [Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue](https://md.research.elamur.ai/free/en/papers/oraby-2016-creating-characterizing-diverse-corpus-sarcasm/) - [Sarcastic or Not: Word Embeddings to Predict the Literal or Sarcastic Meaning of Words](https://md.research.elamur.ai/free/en/papers/ghosh-2015-sarcastic-or-not-word-embeddings/) - [An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews](https://md.research.elamur.ai/free/en/papers/buschmeier-2014-impact-analysis-features-classification-approach/) - [Affective and Contextual Embedding for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/babanejad-2020-affective-contextual-embedding-sarcasm-detection/) - [Sarcasm Analysis Using Conversation Context](https://md.research.elamur.ai/free/en/papers/ghosh-2018-sarcasm-analysis-conversation-context/) - [Your Sentiment Precedes You: Using an author's historical tweets to predict sarcasm](https://md.research.elamur.ai/free/en/papers/khattri-2015-your-sentiment-precedes-you-author/) - [Modelling Irony in Twitter](https://md.research.elamur.ai/free/en/papers/barbieri-2014-modelling-irony-twitter/) - [Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation](https://md.research.elamur.ai/free/en/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/) - [A Report on the 2020 Sarcasm Detection Shared Task](https://md.research.elamur.ai/free/en/papers/ghosh-2020-report-2020-sarcasm-detection-shared/) - [CASCADE: Contextual Sarcasm Detection in Online Discussion Forums](https://md.research.elamur.ai/free/en/papers/hazarika-2018-cascade-contextual-sarcasm-detection-online/) - [R^3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge](https://md.research.elamur.ai/free/en/papers/chakrabarty-2020-r-textasciicircum3-reverse-retrieve-rank/) - [Harnessing Cognitive Features for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/mishra-2016-harnessing-cognitive-features-sarcasm-detection/) - [Towards a Contextual Pragmatic Model to Detect Irony in Tweets](https://md.research.elamur.ai/free/en/papers/karoui-2015-contextual-pragmatic-model-detect-irony/) - [Are you serious?: Rhetorical Questions and Sarcasm in Social Media Dialog](https://md.research.elamur.ai/free/en/papers/oraby-2017-are-you-serious-rhetorical-questions/) - [A Novel Hierarchical BERT Architecture for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/srivastava-2020-novel-hierarchical-bert-architecture-sarcasm/) - [Building a Bridge: A Method for Image-Text Sarcasm Detection Without Pretraining on Image-Text Data](https://md.research.elamur.ai/free/en/papers/wang-2020-building-bridge-method-image-text/) - [iSarcasm: A Dataset of Intended Sarcasm](https://md.research.elamur.ai/free/en/papers/oprea-2020-isarcasm-dataset-intended-sarcasm/) - [Representing Social Media Users for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/kolchinski-2018-representing-social-media-users-sarcasm/) - [DAICT: A Dialectal Arabic Irony Corpus Extracted from Twitter](https://md.research.elamur.ai/free/en/papers/abbes-2020-daict-dialectal-arabic-irony-corpus/) - [A Modular Architecture for Unsupervised Sarcasm Generation](https://md.research.elamur.ai/free/en/papers/mishra-2019-modular-architecture-unsupervised-sarcasm-generation/) - [Modelling Irony in Twitter: Feature Analysis and Evaluation](https://md.research.elamur.ai/free/en/papers/barbieri-2014-modelling-irony-twitter-feature-analysis/) - [We Usually Don't Like Going to the Dentist: Using Common Sense to Detect Irony on Twitter](https://md.research.elamur.ai/free/en/papers/hee-2018-we-usually-don-t-like/) - [Reactive Supervision: A New Method for Collecting Sarcasm Data](https://md.research.elamur.ai/free/en/papers/shmueli-2020-reactive-supervision-new-method-collecting/) - [A Dual-Channel Framework for Sarcasm Recognition by Detecting Sentiment Conflict](https://md.research.elamur.ai/free/en/papers/liu-2022-dual-channel-framework-sarcasm-recognition/) - [Chinese Irony Corpus Construction and Ironic Structure Analysis](https://md.research.elamur.ai/free/en/papers/tang-2014-chinese-irony-corpus-construction-ironic/) - [Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue](https://md.research.elamur.ai/free/en/papers/lukin-2013-really-well-apparently-bootstrapping-improves/) - [“When Numbers Matter!”: Detecting Sarcasm in Numerical Portions of Text](https://md.research.elamur.ai/free/en/papers/dubey-2019-when-numbers-matter-detecting-sarcasm/) - [Prompt Tuning with Contradictory Intentions for Sarcasm Recognition](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-tuning-contradictory-intentions-sarcasm/) - [Getting Reliable Annotations for Sarcasm in Online Dialogues](https://md.research.elamur.ai/free/en/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/) - [Sarcasm Target Identification: Dataset and An Introductory Approach](https://md.research.elamur.ai/free/en/papers/joshi-2018-sarcasm-target-identification-dataset-introductory/) - [X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/han-2022-x-pudu-semeval-2022-task/) - [How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature](https://md.research.elamur.ai/free/en/papers/joshi-2016-how-challenging-is-sarcasm-versus/) - [Interpreting Verbal Irony: Linguistic Strategies and the Connection to the Type of Semantic Incongruity](https://md.research.elamur.ai/free/en/papers/ghosh-2020-interpreting-verbal-irony-linguistic-strategies/) - [YNU-HPCC at SemEval-2022 Task 6: Transformer-based Model for Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/en/papers/zheng-2022-ynu-hpcc-semeval-2022-task/) - [ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets](https://md.research.elamur.ai/free/en/papers/bashmal-2021-arsarcasm-shared-task-ensemble-bert/) - [MarSan at SemEval-2022 Task 6: iSarcasm Detection via T5 and Sequence Learners](https://md.research.elamur.ai/free/en/papers/najafi-2022-marsan-semeval-2022-task-6/) - [Linguistic Features in Automatic Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/ducret-2020-linguistic-features-automatic-sarcasm-detection/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{gonzalezibanez2011identifying, title = {{Identifying Sarcasm in Twitter: A Closer Look}}, author = {González-Ibáñez, Roberto and Muresan, Smaranda and Wacholder, Nina}, booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies}, year = {2011}, url = {https://aclanthology.org/P11-2102/} } ``` - [← previous Rumor has it: Identifying Misinformation in Microblogs](https://md.research.elamur.ai/free/en/papers/qazvinian-2011-rumor-has-it-identifying-misinformation/) - [next → Data-Driven Response Generation in Social Media](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/) - [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/) ## References 14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Carvalho, P., Sarmento, S., Silva, M. J., and de Oliveira, E. 2009. Clues for detecting irony in user-generated contents: oh...!! it's "so easy" ;-). In Proceeding of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion (TSA '09). ACM, New York, NY, USA, 53-56. [find](https://scholar.google.com/scholar?q=Carvalho%2C+P%2E%2C+Sarmento%2C+S%2E%2C+Silva%2C+M%2E+J%2E%2C+and+de+Oliveira%2C+E%2E+2009%2E+Clues+for+detecting+irony+in+user%2Dgenerated+contents%3A+oh%2E%2E%2E%21%21+it%27s+%22so+easy%22+%3B%2D%29%2E+In+Proceeding+of+the+1st+international+CIKM+worksh) 2. Clark, H. and Gerrig, R. 1984. On the pretence theory of irony. Journal of Experimental Psychology: General, 113:121–126. D.C. [find](https://scholar.google.com/scholar?q=Clark%2C+H%2E+and+Gerrig%2C+R%2E+1984%2E+On+the+pretence+theory+of+irony%2E+Journal+of+Experimental+Psychology%3A+General%2C+113%3A121%E2%80%93126%2E+D%2EC%2E) 3. Davidov, D., Tsur, O., and Rappoport, A. 2010. Semi-Supervised Recognition of Sarcastic Sentences in Twitter and Amazon, Dmitry Proceeding of Computational Natural Language Learning (ACL-CoNLL). [find](https://scholar.google.com/scholar?q=Davidov%2C+D%2E%2C+Tsur%2C+O%2E%2C+and+Rappoport%2C+A%2E+2010%2E+Semi%2DSupervised+Recognition+of+Sarcastic+Sentences+in+Twitter+and+Amazon%2C+Dmitry+Proceeding+of+Computational+Natural+Language+Learning+%28ACL%2DCoNLL%29%2E) 4. Derks, D., Bos, A. E. R., and Grumbkow, J. V. 2008. Emoticons and Online Message Interpretation. Soc. Sci. Comput. Rev., 26(3), 379-388. [find](https://scholar.google.com/scholar?q=Derks%2C+D%2E%2C+Bos%2C+A%2E+E%2E+R%2E%2C+and+Grumbkow%2C+J%2E+V%2E+2008%2E+Emoticons+and+Online+Message+Interpretation%2E+Soc%2E+Sci%2E+Comput%2E+Rev%2E%2C+26%283%29%2C+379%2D388%2E) 5. Gibbs, R. 1986. On the psycholinguistics of sarcasm. Journal of Experimental Psychology: General, 105:3–15. [find](https://scholar.google.com/scholar?q=Gibbs%2C+R%2E+1986%2E+On+the+psycholinguistics+of+sarcasm%2E+Journal+of+Experimental+Psychology%3A+General%2C+105%3A3%E2%80%9315%2E) 6. Gibbs, R. W. and Colston H. L. eds. 2007. Irony in Language and Thought. Routledge (Taylor and Francis), New York. [find](https://scholar.google.com/scholar?q=Gibbs%2C+R%2E+W%2E+and+Colston+H%2E+L%2E+eds%2E+2007%2E+Irony+in+Language+and+Thought%2E+Routledge+%28Taylor+and+Francis%29%2C+New+York%2E) 7. Kreuz, R. J. and Glucksberg, S. 1989. How to be sarcastic: The echoic reminder theory of verbal irony. Journal of Experimental Psychology: General, 118:374-386. [find](https://scholar.google.com/scholar?q=Kreuz%2C+R%2E+J%2E+and+Glucksberg%2C+S%2E+1989%2E+How+to+be+sarcastic%3A+The+echoic+reminder+theory+of+verbal+irony%2E+Journal+of+Experimental+Psychology%3A+General%2C+118%3A374%2D386%2E) 8. Kreuz, R. J. and Caucci, G. M. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on Computational Approaches to Figurative Language (pp. 1-4). Rochester, New York: Association for Computational. [find](https://scholar.google.com/scholar?q=Kreuz%2C+R%2E+J%2E+and+Caucci%2C+G%2E+M%2E+2007%2E+Lexical+influences+on+the+perception+of+sarcasm%2E+In+Proceedings+of+the+Workshop+on+Computational+Approaches+to+Figurative+Language+%28pp%2E+1%2D4%29%2E+Rochester%2C+New+York%3A+) [our card](https://md.research.elamur.ai/free/en/papers/kreuz-2007-lexical-influences-perception-sarcasm/) 9. LIWC Inc. 2007. The LIWC application. Retrieved May 10, 2010, from [http://www.liwc.net/liwcdescription.php](http://www.liwc.net/liwcdescription.php). [↗](http://www.liwc.net/liwcdescription.php) 10. Nigam, K. and Hurst, M. 2006. Towards a Robust Metric of Polarity. In Computing Attitude and Affect in Text: Theory and Applications (pp. 265-279). Retrieved February 22, 2010, from http://dx.doi.org/10.1007/1-4020-4102-0_20. [↗](https://doi.org/10.1007/1-4020-4102-0_20) 11. Pak, A. and Paroubek, P. 2010. Twitter as a Corpus for Sentiment Analysis and Opinion Mining, in 'Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)' , European Language Resources Association (ELRA), Valletta, Malta Pang, B. and Lee, L. 2008. Opinion Mining and Sentiment Analysis. Now Publishers Inc, July. [find](https://scholar.google.com/scholar?q=Pak%2C+A%2E+and+Paroubek%2C+P%2E+2010%2E+Twitter+as+a+Corpus+for+Sentiment+Analysis+and+Opinion+Mining%2C+in+%27Proceedings+of+the+Seventh+conference+on+International+Language+Resources+and+Evaluation+%28LREC%2710%29%27+%2C+) [our card](https://md.research.elamur.ai/free/en/papers/pak-2010-twitter-as-corpus-sentiment-analysis/) 12. Pennebaker, J.W., Francis, M.E., & Booth, R.J. (2001). Linguistic Inquiry and Word Count (LIWC): LIWC2001 (this includes the manual only). Mahwah, NJ: Erlbaum Publishers Strapparava, C. and Valitutti, A. 2004. Wordnet-affect: an affective extension of wordnet. In Proceedings of the 4th International Conference on Language Resources and Evaluation, Lisbon. [find](https://scholar.google.com/scholar?q=Pennebaker%2C+J%2EW%2E%2C+Francis%2C+M%2EE%2E%2C+%26+Booth%2C+R%2EJ%2E+%282001%29%2E+Linguistic+Inquiry+and+Word+Count+%28LIWC%29%3A+LIWC2001+%28this+includes+the+manual+only%29%2E+Mahwah%2C+NJ%3A+Erlbaum+Publishers+Strapparava%2C+C%2E+and+Valitutti%2C) [our card](https://md.research.elamur.ai/free/en/papers/strapparava-2004-wordnet-affect-affective-extension-wordnet/) 13. Tepperman, J., Traum, D., and Narayanan, S. 2006. Yeah right: Sarcasm recognition for spoken dialogue systems. In InterSpeech ICSLP, Pittsburgh, PA. [find](https://scholar.google.com/scholar?q=Tepperman%2C+J%2E%2C+Traum%2C+D%2E%2C+and+Narayanan%2C+S%2E+2006%2E+Yeah+right%3A+Sarcasm+recognition+for+spoken+dialogue+systems%2E+In+InterSpeech+ICSLP%2C+Pittsburgh%2C+PA%2E) 14. Utsumi, A. 2000. Verbal irony as implicit display of ironic environment: Distinguishing ironic utterances from nonirony. Journal of Pragmatics, 32(12):1777– 1806. [find](https://scholar.google.com/scholar?q=Utsumi%2C+A%2E+2000%2E+Verbal+irony+as+implicit+display+of+ironic+environment%3A+Distinguishing+ironic+utterances+from+nonirony%2E+Journal+of+Pragmatics%2C+32%2812%29%3A1777%E2%80%93+1806%2E)