---
title: "SemEval-2015 Task 11 — paper by Ghosh et al. (2015)"
description: "This SemEval-2015 shared task asks how well sentiment analysis works on tweets… 70% accuracy. 13 references, cited by 63 papers here."
authors: ["Aniruddha Ghosh", "Guofu Li", "Tony Veale", "Paolo Rosso", "Ekaterina Shutova", "John Barnden", "Antonio Reyes"]
year: 2015
venue: "Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)"
source: "https://aclanthology.org/S15-2080/"
doi: "10.18653/v1/s15-2080"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ghosh-2015-semeval-2015-task-11-sentiment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghosh-2015-semeval-2015-task-11-sentiment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter

# SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter

**Authors:** [Aniruddha Ghosh](https://md.research.elamur.ai/free/en/authors/aniruddha-ghosh/), Guofu Li, [Tony Veale](https://md.research.elamur.ai/free/en/authors/tony-veale/), [Paolo Rosso](https://md.research.elamur.ai/free/en/authors/paolo-rosso/), [Ekaterina Shutova](https://md.research.elamur.ai/free/en/authors/ekaterina-shutova/), John Barnden and [Antonio Reyes](https://md.research.elamur.ai/free/en/authors/antonio-reyes/) · **Year:** 2015 · **Venue:** [Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2015](https://md.research.elamur.ai/free/en/venues/semeval/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S15-2080/) · [PDF](https://aclanthology.org/S15-2080.pdf) · [DOI 10.18653/v1/s15-2080](https://doi.org/10.18653/v1/s15-2080) · [OpenAlex](https://openalex.org/W2101217916)

**Citations:** 206 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/ghosh-2015-semeval-2015-task-11-sentiment/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This SemEval-2015 shared task asks how well sentiment analysis works on tweets that are rich in irony, sarcasm and metaphor. In such tweets, the literal wording often signals a different polarity than the meaning the author intends. Systems were not asked to detect which figure of speech a tweet uses. Instead, they had to give every tweet a detailed sentiment rating ranging from -5 to +5, which covers 11 possible values.

The organisers harvested candidate tweets with the Twitter4j API, using figurative hashtags and marker words. They then widened the seed hashtags with an LSA model. The resulting tweets were annotated on CrowdFlower, with spam filtering and reliability-weighted score aggregation. Submissions were scored by cosine similarity and by mean squared error against the gold scores.

The best team, ClaC, reached a cosine of 0.758. Across systems, sarcastic and ironic tweets proved fairly tractable, while metaphorical and "other" tweets remained much harder. The released data gives researchers a benchmark for studying how figurative language interacts with sentiment.

## Key points
- **Datasets:**
  - A trial set of 1025 tweets came from users known for figurative style, such as comedians. Seven in-house annotators labelled it, and native English speakers' scores counted double.
  - The training and test sets contain 8000 and 4000 tweets, annotated on CrowdFlower.
- **Collection:** Tweets were gathered in June 2014 using hashtags like #sarcasm, #irony, #not and #yeahright, plus words such as "literally" or "figuratively." URLs were stripped, but hashtags were kept. Very short tweets were discarded.
- **Spam control:**
  - Trial tweets served as hidden test questions, and workers below 70% accuracy on them were removed.
  - Workers whose scores never varied (zero standard deviation) were flagged as scammers.
  - So were workers with 60% or more outlier judgements.
  - After cleaning, each tweet kept 5–7 annotations.
- **Results:** ClaC ranked first (cosine 0.758, MSE 2.117), followed by UPF (0.711) and LLT_PolyU (0.687). Sarcasm and irony cosines were roughly 0.87–0.92 for top systems, while metaphor scores were mostly under 0.66.
- **Evaluation:** Systems were ranked with two metrics, cosine similarity and MSE. The detailed results cover 15 teams and their multiple runs.

## Interesting ideas
- **Measure figurativeness extrinsically.** Rather than labelling which device a tweet uses, the task scores sentiment on tweets where figurative usage is common. This avoids disputes over fuzzy category boundaries and could apply to any phenomenon that is hard to define, such as hate speech subtypes.
- **Collect data with cheap signals.** Self-tagging hashtags and marker words act as weak labels for gathering candidate data, and a semantic model (here LSA) expands the seed query set. This is a cheap bootstrap for collecting low-resource or niche corpora, for example Ukrainian social media.
- **Weight annotators by reliability.** Each annotator gets a reliability score equal to their share of in-range answers on gold test questions. The final label is a weighted average using those scores, which is more principled than a plain mean when crowd quality varies.
- **Filter spam on several fronts.** The scheme combines gold-question accuracy, zero-variance detection, and per-item outlier rates relative to the item's spread. Together these give a reusable recipe for cleaning crowdsourced ordinal labels.
- **Report results per subset.** Splitting scores by sarcasm, irony, metaphor and other shows where systems actually fail, which a single aggregate number would hide. This makes the approach useful for diagnosing any benchmark with a mixed composition.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations
- [Universitat Politècnica de València](https://md.research.elamur.ai/free/en/universities/universitat-politecnica-de-valencia/) — University, Valencia, Spain · 21 works · 3,275 citations
- [University College Dublin](https://md.research.elamur.ai/free/en/universities/university-college-dublin/) — University, Dublin, Ireland · 10 works · 1,520 citations
- [University of Birmingham](https://md.research.elamur.ai/free/en/universities/university-of-birmingham/) — University, Birmingham, United Kingdom · 11 works · 62,331 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works · [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) · 155 works

## Cite

Cite as: Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden and Antonio Reyes. 2015. SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ghosh2015semeval,
  title = {{SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter}},
  author = {Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden and Antonio Reyes},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2080},
  url = {https://doi.org/10.18653/v1/s15-2080}
}
```

- [← previous Cross-lingual Dependency Parsing Based on Distributed Representations](https://md.research.elamur.ai/free/en/papers/guo-2015-cross-lingual-dependency-parsing-based/)
- [next → A Transition-based Algorithm for AMR Parsing](https://md.research.elamur.ai/free/en/papers/wang-2015-transition-based-algorithm-amr-parsing/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Barnden, J.A. (2008). Metaphor and artificial intelligence: Why they matter to each other. In R.W. Gibbs, Jr. (Ed.), The Cambridge Handbook of Metaphor and Thought, 311-338. Cambridge, U.K.: Cambridge University Press. [find](https://scholar.google.com/scholar?q=Barnden%2C+J%2EA%2E+%282008%29%2E+Metaphor+and+artificial+intelligence%3A+Why+they+matter+to+each+other%2E+In+R%2EW%2E+Gibbs%2C+Jr%2E+%28Ed%2E%29%2C+The+Cambridge+Handbook+of+Metaphor+and+Thought%2C+311%2D338%2E+Cambridge%2C+U%2EK%2E%3A+Cambridge)
2. Hao, Y., Veale, T. (2010). An Ironic Fist in a Velvet Glove: Creative Mis-Representation in the Construction of Ironic Similes. Minds and Machines 20(4):635–650. [find](https://scholar.google.com/scholar?q=Hao%2C+Y%2E%2C+Veale%2C+T%2E+%282010%29%2E+An+Ironic+Fist+in+a+Velvet+Glove%3A+Creative+Mis%2DRepresentation+in+the+Construction+of+Ironic+Similes%2E+Minds+and+Machines+20%284%29%3A635%E2%80%93650%2E)
3. Li G., Ghosh A., Veale T. (2014). Constructing A corpus of Figurative Language for a Tweet Classification and Retrieval Task. In the preceedings of FIRE 2014, the 6th Forum for Information Retrieval Evaluatio. Bengaluru, India. [find](https://scholar.google.com/scholar?q=Li+G%2E%2C+Ghosh+A%2E%2C+Veale+T%2E+%282014%29%2E+Constructing+A+corpus+of+Figurative+Language+for+a+Tweet+Classification+and+Retrieval+Task%2E+In+the+preceedings+of+FIRE+2014%2C+the+6th+Forum+for+Information+Retrieval+E)
4. Poria, S., Cambria, E., Winterstein, G. and Huang, G.B. (2014). Sentic patterns Dependency-based rules for concept-level sentiment analysis. Knowledge-Based Systems 69, pp. 45-63. [find](https://scholar.google.com/scholar?q=Poria%2C+S%2E%2C+Cambria%2C+E%2E%2C+Winterstein%2C+G%2E+and+Huang%2C+G%2EB%2E+%282014%29%2E+Sentic+patterns+Dependency%2Dbased+rules+for+concept%2Dlevel+sentiment+analysis%2E+Knowledge%2DBased+Systems+69%2C+pp%2E+45%2D63%2E)
5. Reyes A., Rosso P. (2014). On the Difficulty of Automatically Detecting Irony: Beyond a Simple Case of Negation. Knowledge and Information Systems. 40(3): 595-614. DOI: 10.1007/s10115-013-0652-8. — [doi:10.1007/s10115-013-0652-8](https://doi.org/10.1007/s10115-013-0652-8) [find](https://scholar.google.com/scholar?q=Reyes+A%2E%2C+Rosso+P%2E+%282014%29%2E+On+the+Difficulty+of+Automatically+Detecting+Irony%3A+Beyond+a+Simple+Case+of+Negation%2E+Knowledge+and+Information+Systems%2E+40%283%29%3A+595%2D614%2E+DOI%3A+10%2E1007%2Fs10115%2D013%2D0652%2D8%2E+%E2%80%94+%5Bd)
6. Reyes A., Rosso P., Veale T. (2013). A Multidimensional Approach for Detecting Irony in Twitter. Languages Resources and Evaluation 47(1): 239-268. [find](https://scholar.google.com/scholar?q=Reyes+A%2E%2C+Rosso+P%2E%2C+Veale+T%2E+%282013%29%2E+A+Multidimensional+Approach+for+Detecting+Irony+in+Twitter%2E+Languages+Resources+and+Evaluation+47%281%29%3A+239%2D268%2E)
7. Reyes A., Rosso P. (2012). Making Objective Decisions from Subjective Data: Detecting Irony in Customers Reviews. Journal on Decision Support Systems 53(4): 754–760. [find](https://scholar.google.com/scholar?q=Reyes+A%2E%2C+Rosso+P%2E+%282012%29%2E+Making+Objective+Decisions+from+Subjective+Data%3A+Detecting+Irony+in+Customers+Reviews%2E+Journal+on+Decision+Support+Systems+53%284%29%3A+754%E2%80%93760%2E)
8. Reyes A., Rosso P., Buscaldi D. (2012). From Humor Recognition to Irony Detection: The Figurative Language of Social Media. Data & Knowledge Engineering 74:1-12. [find](https://scholar.google.com/scholar?q=Reyes+A%2E%2C+Rosso+P%2E%2C+Buscaldi+D%2E+%282012%29%2E+From+Humor+Recognition+to+Irony+Detection%3A+The+Figurative+Language+of+Social+Media%2E+Data+%26+Knowledge+Engineering+74%3A1%2D12%2E)
9. Shutova, E., L. Sun, A. Korhonen. (2010). [Metaphor identification using verb and noun clustering](https://md.research.elamur.ai/free/en/papers/shutova-2010-metaphor-identification-verb-noun-clustering/). Proceedings of the 23rd International Conference on Computational Linguistics.
10. Veale, T., Keane, M. T. (1992). Conceptual Scaffolding: A spatially founded meaning representation for metaphor comprehension. Computational Intelligence 8(3): 494-519. [find](https://scholar.google.com/scholar?q=Veale%2C+T%2E%2C+Keane%2C+M%2E+T%2E+%281992%29%2E+Conceptual+Scaffolding%3A+A+spatially+founded+meaning+representation+for+metaphor+comprehension%2E+Computational+Intelligence+8%283%29%3A+494%2D519%2E)
11. Veale, T. (2012). Detecting and Generating Ironic Comparisons: An Application of Creative Information Retrieval. [find](https://scholar.google.com/scholar?q=Veale%2C+T%2E+%282012%29%2E+Detecting+and+Generating+Ironic+Comparisons%3A+An+Application+of+Creative+Information+Retrieval%2E)
12. AAAI Fall Symposium Series 2012, Artificial Intelligence of Humor. Arlington, Virginia. [find](https://scholar.google.com/scholar?q=AAAI+Fall+Symposium+Series+2012%2C+Artificial+Intelligence+of+Humor%2E+Arlington%2C+Virginia%2E)
13. Veale, T., Hao, Y. (2007). Comprehending and Generating Apt Metaphors: A Web-driven, Case-based Approach to Figurative Language. In proceedings of AAAI 2007, the 22nd AAAI Conference on Artificial Intelligence. Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Veale%2C+T%2E%2C+Hao%2C+Y%2E+%282007%29%2E+Comprehending+and+Generating+Apt+Metaphors%3A+A+Web%2Ddriven%2C+Case%2Dbased+Approach+to+Figurative+Language%2E+In+proceedings+of+AAAI+2007%2C+the+22nd+AAAI+Conference+on+Artificial+Int)

*Updated 24 October 2026*