---
title: "Really? Well. Apparently Bootstrapping Improves — Lukin 2013"
description: "The paper asks whether a bootstrapping approach built for subjectivity detection in monologic news text (Riloff & Wiebe, R&W) can find sarcasm and…"
authors: ["Stephanie Lukin", "Marilyn Walker"]
year: 2013
venue: "Proceedings of the Workshop on Language Analysis in Social Media"
source: "https://aclanthology.org/W13-1104/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/lukin-2013-really-well-apparently-bootstrapping-improves/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/lukin-2013-really-well-apparently-bootstrapping-improves/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2013](https://md.research.elamur.ai/free/papers/2013/) / Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue

# Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue

**Authors:** [Stephanie Lukin](https://md.research.elamur.ai/free/authors/stephanie-lukin/), [Marilyn Walker](https://md.research.elamur.ai/free/authors/marilyn-walker/) · **Year:** 2013 · **Venue:** Proceedings of the Workshop on Language Analysis in Social Media · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-1104/) · [PDF](https://aclanthology.org/W13-1104.pdf)

**Citations:** [30](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper asks whether a bootstrapping approach built for subjectivity detection in monologic news text (Riloff & Wiebe, R&W) can find sarcasm and nastiness in online debate dialogue, where annotated examples are rare and costly. The authors use Quote/Response pairs from the 4forums portion of the Internet Argument Corpus (IAC), which has Mechanical Turk labels. They first build lexical cue sets in two ways: by statistical selection (χ²) and by having crowd workers pick indicative phrases. These cues feed a high-precision first-stage classifier. The utterances it labels are then used to learn generalized syntactic extraction patterns in R&W's style. A pattern-based classifier built from those patterns is applied to held-out data.

Human-selected cues generalized better, while χ² cues seemed overfitted to the development data. Pattern-based bootstrapping raised both precision and recall for each phenomenon. The study shows that a news-oriented method transfers reasonably well to social dialogue. This points to a route for growing labeled corpora of subjective dialogue acts with little manual annotation.

## Key points
- **Data:** About 10,003 IAC Q/R pairs from 4forums were analyzed.
  - Roughly 12% are sarcastic and 12% are nasty.
  - Seven Turkers rated each pair: sarcasm as a binary judgment, and nasty/nice on a −5 to 5 scale.
  - Sarcasm was taken as a mean above 0.5, nasty as below −1, and nice as above 1.
- **Sarcasm results:** The first-stage classifier with crowdsourced cues reached 54% precision and 38% recall. Bootstrapping with syntactic patterns raised this to 62% precision and 52% recall (F 0.57).
- **Nastiness results:** The first stage gave 58% precision and 49% recall. After pattern bootstrapping, this rose to 75% precision and 62% recall (F 0.69). The authors read this as a sign that insults are easier to detect than sarcasm.
- **χ² cues underperformed:** Pattern classifiers built on statistically selected cues did much worse, for example 31% precision for sarcasm and 30% for nastiness.
- **Future work:** The authors suggest using semantic word categories such as LIWC instead of syntactic ones, running more bootstrapping iterations, and extending the method to other dialogue acts and domains.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

30 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 16 not yet.

- [Modelling Irony in Twitter: Feature Analysis and Evaluation](https://md.research.elamur.ai/free/papers/barbieri-2014-modelling-irony-twitter-feature-analysis/)
- [Modelling Sarcasm in Twitter, a Novel Approach](https://md.research.elamur.ai/free/papers/barbieri-2014-modelling-sarcasm-twitter-novel-approach/)
- [Fracking Sarcasm using Neural Network](https://md.research.elamur.ai/free/papers/ghosh-2016-fracking-sarcasm-neural-network/)
- [Magnets for Sarcasm: Making Sarcasm Detection Timely, Contextual and Very Personal](https://md.research.elamur.ai/free/papers/ghosh-2017-magnets-sarcasm-making-sarcasm-detection/)
- [Automatic Sarcasm Detection: A Survey](https://md.research.elamur.ai/free/papers/joshi-2016-automatic-sarcasm-detection-survey/)
- [How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature](https://md.research.elamur.ai/free/papers/joshi-2016-how-challenging-is-sarcasm-versus/)
- [A Large Self-Annotated Corpus for Sarcasm](https://md.research.elamur.ai/free/papers/khodak-2018-large-self-annotated-corpus-sarcasm/)
- [A Dual-Channel Framework for Sarcasm Recognition by Detecting Sentiment Conflict](https://md.research.elamur.ai/free/papers/liu-2022-dual-channel-framework-sarcasm-recognition/)
- [Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue](https://md.research.elamur.ai/free/papers/oraby-2016-creating-characterizing-diverse-corpus-sarcasm/)
- [Sarcasm as Contrast between a Positive Sentiment and Negative Situation](https://md.research.elamur.ai/free/papers/prag-2013-riloff-sarcasm-as-contrast-between-positive/)
- [Getting Reliable Annotations for Sarcasm in Online Dialogues](https://md.research.elamur.ai/free/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/)
- [Chinese Irony Corpus Construction and Ironic Structure Analysis](https://md.research.elamur.ai/free/papers/tang-2014-chinese-irony-corpus-construction-ironic/)
- [Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?](https://md.research.elamur.ai/free/papers/yao-2024-sarcasm-detection-step-step-reasoning/)
- [Tweet Sarcasm Detection Using Deep Neural Network](https://md.research.elamur.ai/free/papers/zhang-2016-tweet-sarcasm-detection-deep-neural/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lukin2013really,
  title = {{Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue}},
  author = {Lukin, Stephanie and Walker, Marilyn},
  booktitle = {Proceedings of the Workshop on Language Analysis in Social Media},
  year = {2013},
  url = {https://aclanthology.org/W13-1104/}
}
```

- [← previous Ranking Translations using Error Analysis and Quality Estimation](https://md.research.elamur.ai/free/papers/fishel-2013-ranking-translations-error-analysis-quality/)
- [next → Recurrent Convolutional Neural Networks for Discourse Compositionality](https://md.research.elamur.ai/free/papers/kalchbrenner-2013-recurrent-convolutional-neural-networks-discourse/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/papers/2013/)

## References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. G.A. Bryant and J.E. Fox Tree. 2002. Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. [↗](https://doi.org/10.1207/s15327868ms1702_2)
2. P. Carvalho, L. Sarmento, M.J. Silva, and E. de Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-). In Proc. of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, p. 53–56. ACM. [find](https://scholar.google.com/scholar?q=P%2E+Carvalho%2C+L%2E+Sarmento%2C+M%2EJ%2E+Silva%2C+and+E%2E+de+Oliveira%2E+2009%2E+Clues+for+detecting+irony+in+user%2Dgenerated+contents%3A+oh%2E%2E%2E%21%21+it%E2%80%99s+so+easy%3B%2D%29%2E+In+Proc%2E+of+the+1st+international+CIKM+workshop+on+Topic%2D)
3. D. Davidov, O. Tsur, and A. Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proc. of the Fourteenth Conference on Computational Natural Language Learning, p. 107– 116. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=D%2E+Davidov%2C+O%2E+Tsur%2C+and+A%2E+Rappoport%2E+2010%2E+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+twitter+and+amazon%2E+In+Proc%2E+of+the+Fourteenth+Conference+on+Computational+Natural+Language+Learning%2C) [our card](https://md.research.elamur.ai/free/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
4. J. Eisterhold, S. Attardo, and D. Boxer. 2006. Reactions to irony in discourse: Evidence for the least disruption principle. Journal of Pragmatics, 38(8):1239–1256. [↗](https://doi.org/10.1016/j.pragma.2004.12.003)
5. E. Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Language Resources and Evaluation Conference, LREC2012. [find](https://scholar.google.com/scholar?q=E%2E+Filatova%2E+2012%2E+Irony+and+sarcasm%3A+Corpus+generation+and+analysis+using+crowdsourcing%2E+In+Language+Resources+and+Evaluation+Conference%2C+LREC2012%2E) [our card](https://md.research.elamur.ai/free/papers/filatova-2012-irony-sarcasm-corpus-generation-analysis/)
6. J.E. Fox Tree and J.C. Schrock. 1999. Discourse Markers in Spontaneous Speech: Oh What a Difference an Oh Makes. Journal of Memory and Language, 40(2):280–295. [↗](https://doi.org/10.1006/jmla.1998.2613)
7. J. E. Fox Tree. 2010. Discourse markers across speakers and settings. Language and Linguistics Compass, 3(1):1–13. [↗](https://doi.org/10.1111/j.1749-818x.2010.00195.x)
8. P. Gianfortoni, D. Adamson, and C.P. Rosé. 2011. Modeling of stylistic variation in social media with stretchy patterns. In Proc. of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, p. 49–59. ACL. [find](https://scholar.google.com/scholar?q=P%2E+Gianfortoni%2C+D%2E+Adamson%2C+and+C%2EP%2E+Rose%CC%81%2E+2011%2E+Modeling+of+stylistic+variation+in+social+media+with+stretchy+patterns%2E+In+Proc%2E+of+the+First+Workshop+on+Algorithms+and+Resources+for+Modelling+of+Di)
9. R.W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. [↗](https://doi.org/10.1080/10926488.2000.9678862)
10. R. González-Ibáñez, S. Muresan, and N. Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proc. of the 49th Annual Meeting of the ACL: Human Language Technologies: short papers, volume 2, p. 581–586. [find](https://scholar.google.com/scholar?q=R%2E+Gonza%CC%81lez%2DIba%CC%81n%CC%83ez%2C+S%2E+Muresan%2C+and+N%2E+Wacholder%2E+2011%2E+Identifying+sarcasm+in+twitter%3A+a+closer+look%2E+In+Proc%2E+of+the+49th+Annual+Meeting+of+the+ACL%3A+Human+Language+Technologies%3A+short+papers%2C+vol) [our card](https://md.research.elamur.ai/free/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/)
11. A. Razavi, D. Inkpen, S. Uritsky, and S. Matwin. 2010. Offensive language detection using multi-level classification. Advances in Artificial Intelligence, p. 16–27. [find](https://scholar.google.com/scholar?q=A%2E+Razavi%2C+D%2E+Inkpen%2C+S%2E+Uritsky%2C+and+S%2E+Matwin%2E+2010%2E+Offensive+language+detection+using+multi%2Dlevel+classification%2E+Advances+in+Artificial+Intelligence%2C+p%2E+16%E2%80%9327%2E)
12. A. Reyes and P. Rosso. 2011. Mining subjective knowledge from customer reviews: a specific case of irony detection. In Proc. of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), ACL, p. 118–124. [find](https://scholar.google.com/scholar?q=A%2E+Reyes+and+P%2E+Rosso%2E+2011%2E+Mining+subjective+knowledge+from+customer+reviews%3A+a+specific+case+of+irony+detection%2E+In+Proc%2E+of+the+2nd+Workshop+on+Computational+Approaches+to+Subjectivity+and+Sentime) [our card](https://md.research.elamur.ai/free/papers/reyes-2011-mining-subjective-knowledge-customer-reviews/)
13. A. Reyes, P. Rosso, and D. Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering. [↗](https://doi.org/10.1016/j.datak.2012.02.005)
14. E. Riloff and J. Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proc. of the 2003 conference on Empirical methods in Natural Language Processing-V. 10, p. 105–112. ACL. [find](https://scholar.google.com/scholar?q=E%2E+Riloff+and+J%2E+Wiebe%2E+2003%2E+Learning+extraction+patterns+for+subjective+expressions%2E+In+Proc%2E+of+the+2003+conference+on+Empirical+methods+in+Natural+Language+Processing%2DV%2E+10%2C+p%2E+105%E2%80%93112%2E+ACL%2E)
15. R. Snow, B. O’Conner, D. Jurafsky, and A.Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks In Proc. of the Conference on Empirical Methods in Natural Language Processing, p. 254–263. ACM. [find](https://scholar.google.com/scholar?q=R%2E+Snow%2C+B%2E+O%E2%80%99Conner%2C+D%2E+Jurafsky%2C+and+A%2EY%2E+Ng%2E+2008%2E+Cheap+and+fast%E2%80%94but+is+it+good%3F%3A+evaluating+non%2Dexpert+annotations+for+natural+language+tasks+In+Proc%2E+of+the+Conference+on+Empirical+Methods+in+Na)
16. S.O. Sood, E.F. Churchill, and J. Antin. 2011. Automatic identification of personal insults on social news sites. Journal of the American Society for Information Science and Technology. [↗](https://doi.org/10.1002/asi.21690)
17. Dan Sperber and Deidre Wilson. 1981. Irony and the use-mention distinction. In Peter Cole, editor, Radical Pragmatics, p. 295–318. Academic Press, N.Y. [find](https://scholar.google.com/scholar?q=Dan+Sperber+and+Deidre+Wilson%2E+1981%2E+Irony+and+the+use%2Dmention+distinction%2E+In+Peter+Cole%2C+editor%2C+Radical+Pragmatics%2C+p%2E+295%E2%80%93318%2E+Academic+Press%2C+N%2EY%2E)
18. E. Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. of the National Conference on Artificial Intelligence, p. 1058–1065. [find](https://scholar.google.com/scholar?q=E%2E+Spertus%2E+1997%2E+Smokey%3A+Automatic+recognition+of+hostile+messages%2E+In+Proc%2E+of+the+National+Conference+on+Artificial+Intelligence%2C+p%2E+1058%E2%80%931065%2E)
19. M. Thelen and E. Riloff. 2002. A bootstrapping method for learning semantic lexicons using extraction pattern contexts. In Proc. of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, p. 214–221. ACL. [find](https://scholar.google.com/scholar?q=M%2E+Thelen+and+E%2E+Riloff%2E+2002%2E+A+bootstrapping+method+for+learning+semantic+lexicons+using+extraction+pattern+contexts%2E+In+Proc%2E+of+the+ACL%2D02+conference+on+Empirical+methods+in+natural+language+proce)
20. O. Tsur, D. Davidov, and A. Rappoport. 2010. Icwsm– a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proc. of the fourth international AAAI conference on weblogs and social media, p. 162–169. [find](https://scholar.google.com/scholar?q=O%2E+Tsur%2C+D%2E+Davidov%2C+and+A%2E+Rappoport%2E+2010%2E+Icwsm%E2%80%93+a+great+catchy+name%3A+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+online+product+reviews%2E+In+Proc%2E+of+the+fourth+international+AAAI+confere)
21. Marilyn Walker, Pranav Anand, , Robert Abbott, and Jean E. Fox Tree. 2012. A corpus for research on deliberation and debate. In Language Resources and Evaluation Conference, LREC2012. [find](https://scholar.google.com/scholar?q=Marilyn+Walker%2C+Pranav+Anand%2C+%2C+Robert+Abbott%2C+and+Jean+E%2E+Fox+Tree%2E+2012%2E+A+corpus+for+research+on+deliberation+and+debate%2E+In+Language+Resources+and+Evaluation+Conference%2C+LREC2012%2E) [our card](https://md.research.elamur.ai/free/papers/walker-2012-corpus-research-deliberation-debate/)
22. J.M. Wiebe, R.F. Bruce, and T.P. O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proc. of the 37th annual meeting of the Association for Computational Linguistics, p. 246–253. ACL. [find](https://scholar.google.com/scholar?q=J%2EM%2E+Wiebe%2C+R%2EF%2E+Bruce%2C+and+T%2EP%2E+O%E2%80%99Hara%2E+1999%2E+Development+and+use+of+a+gold%2Dstandard+data+set+for+subjectivity+classifications%2E+In+Proc%2E+of+the+37th+annual+meeting+of+the+Association+for+Computationa)
23. J. Wiebe, E. Breck, C. Buckley, C. Cardie, P. Davis, B. Fraser, D. Litman, D. Pierce, E. Riloff, T. Wilson, et al. 2003. Recognizing and organizing opinions expressed in the world press. In Working Notes-New Directions in Question Answering (AAAI Spring Symposium Series). [find](https://scholar.google.com/scholar?q=J%2E+Wiebe%2C+E%2E+Breck%2C+C%2E+Buckley%2C+C%2E+Cardie%2C+P%2E+Davis%2C+B%2E+Fraser%2C+D%2E+Litman%2C+D%2E+Pierce%2C+E%2E+Riloff%2C+T%2E+Wilson%2C+et+al%2E+2003%2E+Recognizing+and+organizing+opinions+expressed+in+the+world+press%2E+In+Working+No)
24. T. Wilson, P. Hoffmann, S. Somasundaran, J. Kessler, J. Wiebe, Y. Choi, C. Cardie, E. Riloff, and S. Patwardhan. 2005. Opinionfinder: A system for subjectivity analysis. In Proc. of HLT/EMNLP on Interactive Demonstrations, p. 34–35. ACL. [find](https://scholar.google.com/scholar?q=T%2E+Wilson%2C+P%2E+Hoffmann%2C+S%2E+Somasundaran%2C+J%2E+Kessler%2C+J%2E+Wiebe%2C+Y%2E+Choi%2C+C%2E+Cardie%2C+E%2E+Riloff%2C+and+S%2E+Patwardhan%2E+2005%2E+Opinionfinder%3A+A+system+for+subjectivity+analysis%2E+In+Proc%2E+of+HLT%2FEMNLP+on+Inter)
25. G. Xiang, B. Fan, L. Wang, J. Hong, and C. Rose. 2012. Detecting offensive tweets via topical feature discovery over a large scale twitter corpus. In Proc. of the 21st ACM international conference on Information and knowledge management, p. 1980–1984. ACM. [find](https://scholar.google.com/scholar?q=G%2E+Xiang%2C+B%2E+Fan%2C+L%2E+Wang%2C+J%2E+Hong%2C+and+C%2E+Rose%2E+2012%2E+Detecting+offensive+tweets+via+topical+feature+discovery+over+a+large+scale+twitter+corpus%2E+In+Proc%2E+of+the+21st+ACM+international+conference+on+)