---
title: "Getting Reliable Annotations for Sarcasm in — Swanson 2014"
description: "Sarcasm is common in online forum debates, but collecting trustworthy labels for it is difficult because the phenomenon is subjective and sometimes…"
authors: ["Reid Swanson", "Stephanie Lukin", "Luke Eisenberg", "Thomas Corcoran", "Marilyn Walker"]
year: 2014
venue: "Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)"
source: "https://aclanthology.org/L14-1046/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2014](https://md.research.elamur.ai/free/papers/2014/) / Getting Reliable Annotations for Sarcasm in Online Dialogues

# Getting Reliable Annotations for Sarcasm in Online Dialogues

**Authors:** Reid Swanson, [Stephanie Lukin](https://md.research.elamur.ai/free/authors/stephanie-lukin/), Luke Eisenberg, Thomas Corcoran, [Marilyn Walker](https://md.research.elamur.ai/free/authors/marilyn-walker/) · **Year:** 2014 · **Venue:** Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L14-1046/) · [PDF](https://aclanthology.org/L14-1046.pdf) · [OpenAlex](https://openalex.org/W2962739734)

**Citations:** 15 citations (openalex-snapshot · 2026-10-14) · [8](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Sarcasm is common in online forum debates, but collecting trustworthy labels for it is difficult because the phenomenon is subjective and sometimes intentionally ambiguous. The authors study how to obtain dependable sarcasm judgments from untrained Mechanical Turk workers on Quote-Response pairs from the Internet Argument Corpus (IAC). They gathered up to 25 crowd annotations for 600 utterances, including items whose status was initially unclear. They then compared several ways of deriving a final label: Majority Class, Kappa-based weighting, EM (Dawid–Skene style), and Karger's iterative algorithm. They also tracked how labels shift as more annotators are added. The more sophisticated worker-reliability models gave no clear advantage over simple majority voting or vote-count thresholds. In most cases, labels settled after roughly 7–10 annotators. This gives practical guidance on annotation budgets for subjective dialogue phenomena. It also prompts a re-examination of the earlier two-vote threshold used in their own previous sarcasm classifier work (L&W).

## Key points
- **Data source:** The original IAC sarcasm annotation covered 10,003 Q-R pairs, with 5–7 Turkers per item and Krippendorff's α of only 0.23. The new, larger study reached α = 0.387.
- **Earlier threshold:** The previous work (Lukin & Walker 2013) labelled a post SARC if at least 2 annotators marked it sarcastic. Karger's labelling proved considerably more conservative than this rule.
- **Accuracy on ambiguous items:** For the 400 initially ambiguous items, accuracy against the final Karger labels was about 80% with 3 annotators and close to 90% with 10. Adding annotators beyond 10 brought only small gains.
- **Remaining ambiguity:** After 25 annotations, only about 6–9 of the 600 utterances lacked a clear majority across the four methods.
- **Recommendation:** Collect 3 annotations per item, and extend to about 7 when the first judgments disagree strongly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 3 not yet.

- [Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations](https://md.research.elamur.ai/free/papers/abercrombie-2016-putting-sarcasm-detection-into-context/)
- [A Large Self-Annotated Corpus for Sarcasm](https://md.research.elamur.ai/free/papers/khodak-2018-large-self-annotated-corpus-sarcasm/)
- [Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue](https://md.research.elamur.ai/free/papers/oraby-2016-creating-characterizing-diverse-corpus-sarcasm/)
- [Are you serious?: Rhetorical Questions and Sarcasm in Social Media Dialog](https://md.research.elamur.ai/free/papers/oraby-2017-are-you-serious-rhetorical-questions/)
- [Reactive Supervision: A New Method for Collecting Sarcasm Data](https://md.research.elamur.ai/free/papers/shmueli-2020-reactive-supervision-new-method-collecting/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{swanson2014getting,
  title = {{Getting Reliable Annotations for Sarcasm in Online Dialogues}},
  author = {Swanson, Reid and Lukin, Stephanie and Eisenberg, Luke and Corcoran, Thomas and Walker, Marilyn},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5g5wpyeg8tiv},
  url = {https://doi.org/10.63317/5g5wpyeg8tiv}
}
```

- [← previous FrameNet: A Knowledge Base for Natural Language Processing](https://md.research.elamur.ai/free/papers/baker-2014-framenet-knowledge-base-natural-language/)
- [next → Grounded Compositional Semantics for Finding and Describing Images with Sentences](https://md.research.elamur.ai/free/papers/socher-2014-grounded-compositional-semantics-finding-describing/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/papers/2014/)

## References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Brown, P. and Levinson, S. (1987). Politeness: Some universals in language usage. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E+and+Levinson%2C+S%2E+%281987%29%2E+Politeness%3A+Some+universals+in+language+usage%2E+Cambridge+University+Press%2E) [our card](https://md.research.elamur.ai/free/papers/brown-1987-politeness-some-universals-language-usage/)
2. Bryant, G. and Fox Tree, J. (2002). Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. [↗](https://doi.org/10.1207/s15327868ms1702_2)
3. Callison-Burch, C. (2009). Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proc. of the 2009 Conf. on Empirical Methods in Natural Language Processing, p. 286–295. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E+%282009%29%2E+Fast%2C+cheap%2C+and+creative%3A+Evaluating+translation+quality+using+Amazon%E2%80%99s+Mechanical+Turk%2E+In+Proc%2E+of+the+2009+Conf%2E+on+Empirical+Methods+in+Natural+Language+Processing%2C+p%2E+) [our card](https://md.research.elamur.ai/free/papers/burch-2009-fast-cheap-creative-evaluating-translation/)
4. Dagan, I., Glickman, O., and Magnini, B. (2006). The PASCAL recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, Lecture Notes in Computer Science 3944, p. 177–190. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+and+Magnini%2C+B%2E+%282006%29%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classification%2C+and+)
5. Davidov, D., Tsur, O., and Rappoport, A. (2010). Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proc. of the Fourteenth Conf. on Computational Natural Language Learning, p. 107–116. [find](https://scholar.google.com/scholar?q=Davidov%2C+D%2E%2C+Tsur%2C+O%2E%2C+and+Rappoport%2C+A%2E+%282010%29%2E+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+twitter+and+amazon%2E+In+Proc%2E+of+the+Fourteenth+Conf%2E+on+Computational+Natural+Language+Learning%2C+) [our card](https://md.research.elamur.ai/free/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
6. Dawid, A. P. and Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, p. 20–28. [↗](https://doi.org/10.2307/2346806)
7. Di Eugenio, B. and Glass, M. (2004). The kappa statistic: A second look. Computational linguistics, 30(1):95– 101. [↗](https://doi.org/10.1162/089120104773633402)
8. Filatova, E. (2012). Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Language Resources and Evaluation Conf. , LREC2012. [find](https://scholar.google.com/scholar?q=Filatova%2C+E%2E+%282012%29%2E+Irony+and+sarcasm%3A+Corpus+generation+and+analysis+using+crowdsourcing%2E+In+Language+Resources+and+Evaluation+Conf%2E+%2C+LREC2012%2E) [our card](https://md.research.elamur.ai/free/papers/filatova-2012-irony-sarcasm-corpus-generation-analysis/)
9. Gibbs, R. (2000). Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. [↗](https://doi.org/10.1080/10926488.2000.9678862)
10. González-Ibáñez, R., Muresan, S., and Wacholder, N. (2011). Identifying sarcasm in twitter: a closer look. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics, volume 2, p. 581–586. [find](https://scholar.google.com/scholar?q=Gonza%CC%81lez%2DIba%CC%81n%CC%83ez%2C+R%2E%2C+Muresan%2C+S%2E%2C+and+Wacholder%2C+N%2E+%282011%29%2E+Identifying+sarcasm+in+twitter%3A+a+closer+look%2E+In+Proc%2E+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+volu) [our card](https://md.research.elamur.ai/free/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/)
11. Karger, D. R., Oh, S., and Shah, D. (2011). Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, p. 1953–1961. [find](https://scholar.google.com/scholar?q=Karger%2C+D%2E+R%2E%2C+Oh%2C+S%2E%2C+and+Shah%2C+D%2E+%282011%29%2E+Iterative+learning+for+reliable+crowdsourcing+systems%2E+In+Advances+in+neural+information+processing+systems%2C+p%2E+1953%E2%80%931961%2E)
12. Levinson, S. C. (1981). Some pre-observations on the modelling of dialogue. Discourse Processes, 4:93–116. [↗](https://doi.org/10.1080/01638538109544510)
13. Levinson, S. C. (1985). What’s special about conversational inference. In 1987 Linguistics Institute Packet. [find](https://scholar.google.com/scholar?q=Levinson%2C+S%2E+C%2E+%281985%29%2E+What%E2%80%99s+special+about+conversational+inference%2E+In+1987+Linguistics+Institute+Packet%2E)
14. Liu, Q., Peng, J., and Ihler, A. (2012). Variational inference for crowdsourcing. Advances in Neural Information Processing Systems 25, p. 701–709. [find](https://scholar.google.com/scholar?q=Liu%2C+Q%2E%2C+Peng%2C+J%2E%2C+and+Ihler%2C+A%2E+%282012%29%2E+Variational+inference+for+crowdsourcing%2E+Advances+in+Neural+Information+Processing+Systems+25%2C+p%2E+701%E2%80%93709%2E)
15. Louis, A. and Nenkova, A. (2011). Automatic identification of general and specific sentences by leveraging discourse annotations. In IJCNLP, p. 605–613. [find](https://scholar.google.com/scholar?q=Louis%2C+A%2E+and+Nenkova%2C+A%2E+%282011%29%2E+Automatic+identification+of+general+and+specific+sentences+by+leveraging+discourse+annotations%2E+In+IJCNLP%2C+p%2E+605%E2%80%93613%2E)
16. Lukin, S. and Walker, M. (2013). Really? Well. Apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. [find](https://scholar.google.com/scholar?q=Lukin%2C+S%2E+and+Walker%2C+M%2E+%282013%29%2E+Really%3F+Well%2E+Apparently+bootstrapping+improves+the+performance+of+sarcasm+and+nastiness+classifiers+for+online+dialogue%2E) [our card](https://md.research.elamur.ai/free/papers/lukin-2013-really-well-apparently-bootstrapping-improves/)
17. Language and Social Media Workshop, NAACL 2013, p. 30. [find](https://scholar.google.com/scholar?q=Language+and+Social+Media+Workshop%2C+NAACL+2013%2C+p%2E+30%2E)
18. Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., and Huang, R. (2013). Sarcasm as contrast between a positive sentiment and negative situation. [find](https://scholar.google.com/scholar?q=Riloff%2C+E%2E%2C+Qadir%2C+A%2E%2C+Surve%2C+P%2E%2C+De+Silva%2C+L%2E%2C+Gilbert%2C+N%2E%2C+and+Huang%2C+R%2E+%282013%29%2E+Sarcasm+as+contrast+between+a+positive+sentiment+and+negative+situation%2E) [our card](https://md.research.elamur.ai/free/papers/prag-2013-riloff-sarcasm-as-contrast-between-positive/)
19. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. (2008). Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proc. of the Conf. on Empirical Methods in Natural Language Processing, p. 254–263. [find](https://scholar.google.com/scholar?q=Snow%2C+R%2E%2C+O%E2%80%99Connor%2C+B%2E%2C+Jurafsky%2C+D%2E%2C+and+Ng%2C+A%2E+%282008%29%2E+Cheap+and+fast%E2%80%94but+is+it+good%3F%3A+evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proc%2E+of+the+Conf%2E+on+Empirical+Methods+in+Nat)
20. Walker, M., Anand, P., , Abbott, R., and Fox Tree, J. E. (2012). A corpus for research on deliberation and debate. In Language Resources and Evaluation Conf. , LREC2012. [find](https://scholar.google.com/scholar?q=Walker%2C+M%2E%2C+Anand%2C+P%2E%2C+%2C+Abbott%2C+R%2E%2C+and+Fox+Tree%2C+J%2E+E%2E+%282012%29%2E+A+corpus+for+research+on+deliberation+and+debate%2E+In+Language+Resources+and+Evaluation+Conf%2E+%2C+LREC2012%2E) [our card](https://md.research.elamur.ai/free/papers/walker-2012-corpus-research-deliberation-debate/)