---
title: "SemEval-2015 Task 2 — Agirre 2015"
description: "This paper reports on the 2015 edition of the SemEval shared task on semantic textual similarity (STS)."
authors: ["Eneko Agirre", "Carmen Banea", "Claire Cardie", "Daniel Cer", "Mona Diab", "Aitor Gonzalez-Agirre", "Weiwei Guo", "Iñigo Lopez-Gazpio", "Montse Maritxalar", "Rada Mihalcea", "German Rigau", "Larraitz Uria", "Janyce Wiebe"]
year: 2015
venue: "Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)"
source: "https://aclanthology.org/S15-2045/"
doi: "10.18653/v1/s15-2045"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

# SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

**Authors:** [Eneko Agirre](https://md.research.elamur.ai/free/en/authors/eneko-agirre/), [Carmen Banea](https://md.research.elamur.ai/free/en/authors/carmen-banea/), [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/), [Daniel Cer](https://md.research.elamur.ai/free/en/authors/daniel-cer/), [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/), [Aitor Gonzalez-Agirre](https://md.research.elamur.ai/free/en/authors/aitor-gonzalez-agirre/), [Weiwei Guo](https://md.research.elamur.ai/free/en/authors/weiwei-guo/), Iñigo Lopez-Gazpio, Montse Maritxalar, [Rada Mihalcea](https://md.research.elamur.ai/free/en/authors/rada-mihalcea/), [German Rigau](https://md.research.elamur.ai/free/en/authors/german-rigau/), Larraitz Uria et al. (13 authors) · **Year:** 2015 · **Venue:** Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S15-2045/) · [PDF](https://aclanthology.org/S15-2045.pdf) · [DOI 10.18653/v1/s15-2045](https://doi.org/10.18653/v1/s15-2045) · [OpenAlex](https://openalex.org/W2133458109)

**Citations:** 466 citations (openalex-snapshot · 2026-10-14) · [292](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This paper reports on the 2015 edition of the SemEval shared task on semantic textual similarity (STS). In STS, a system assigns a graded, symmetric score showing how close in meaning two short texts are. The organizers built new evaluation data in English and Spanish. The English data includes news headlines (HDL), image captions (Images), tutoring-system answers (Answers-student), Stack Exchange answers (Answers-forum) and DEFT discussion-forum sentences (Belief). The Spanish data adds news and Wikipedia pairs. Candidate pairs were sampled by string similarity, labelled by crowd workers and then filtered automatically.

A new pilot subtask on interpretable STS asked systems to explain their scores. Systems had to align chunks across the two sentences, label each alignment with a relation type and give each aligned pair its own score. Participation was strong:
- 29 teams (74 runs) in English;
- 7 teams (16 runs) in Spanish;
- 7 teams (29 runs) in the pilot.

In the pilot, systems did clearly worse when they had to find chunks themselves than when gold chunks were provided. The task moves STS toward explainable similarity judgments that are useful for MT evaluation, QA and summarization.

## Key points
- English test sets: HDL, Images and Answers-student have 750 pairs each; Answers-forum and Belief have 375 each. Training data came from the 2012–2014 STS releases (MSRpar, MSRvid, OnWN, SMTnews, Deft-forum, Tweet-news, etc.).
- Scoring scales: English uses 0–5. Spanish uses 0–4, with the English grades 3 and 4 merged into a single grade.
- English annotation setup on Amazon Mechanical Turk:
  - only Master-qualified workers;
  - 5 pairs per HIT at $0.20;
  - 5 labels per pair;
  - 105 hidden gold pairs from STS 2013, used to detect and remove poor annotators.
- Interpretable subtask design:
  - run on Headlines and Images;
  - two scenarios, gold chunks and system chunks;
  - F-measures for alignment, type, score and type+score.
- Interpretable subtask results:
  - Images was generally harder than Headlines. With gold chunks, the average F Type+Score was 0.4748 on Images versus 0.5381 on Headlines.
  - In the system-chunks scenario, the UMDuluth BlueTeam runs ranked first on both Headlines and Images.

## Interesting ideas
- **Balanced sampling by string similarity:** Candidate pairs were drawn evenly across string-similarity ranges, both from same-cluster and different-cluster sources. This gives a dataset that covers the whole similarity scale instead of piling up near zero. It is cheap to reuse when building any graded-similarity benchmark, for example a Ukrainian one.
- **Automatic filtering against three criteria:** After annotation, pairs were selected purely automatically to balance three goals: an even score spread, high annotator agreement, and difficulty for a string-matching baseline. The release of the scripts and raw labels makes the final gold standard reproducible.
- **Gold items used only for after-the-fact cleanup:** Hidden gold pairs were not used to train or auto-ban workers. They served only to identify and discard bad annotators' data later. This is a softer quality-control scheme that avoids discouraging skilled crowd workers.
- **Agreement measured like system evaluation:** Annotator quality was estimated by correlating each annotator with the average of the others. Because this matches how systems are scored, it doubles as a rough human upper bound.
- **Explanations evaluated as a layered alignment:** Interpretability was operationalized as chunk alignment plus relation type plus per-chunk score, each with its own F-measure. Comparing a gold-chunk scenario with a system-chunk scenario separates segmentation errors from alignment errors, which is a useful diagnostic for any explainable-NLP evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

292 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 275 not yet.

- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/)
- [ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation](https://md.research.elamur.ai/free/en/papers/huang-2023-paraamr-large-scale-syntactically-diverse/)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/)
- [A Continuously Growing Dataset of Sentential Paraphrases](https://md.research.elamur.ai/free/en/papers/lan-2017-continuously-growing-dataset-sentential-paraphrases/)
- [On the Sentence Embeddings from Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/)
- [Explainable Semantic Textual Similarity via Dissimilar Span Detection](https://md.research.elamur.ai/free/en/papers/lozano-2026-explainable-semantic-textual-similarity-dissimilar/)
- [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/)
- [Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/)
- [Edition 2.0 of the PARSEME shared task on multilingual identification and paraphrasing of multiword expressions](https://md.research.elamur.ai/free/en/papers/scholivet-2026-edition-2-0-parseme-shared/)
- [ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations](https://md.research.elamur.ai/free/en/papers/wieting-2018-paranmt-50m-pushing-limits-paraphrastic/)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://md.research.elamur.ai/free/en/papers/yan-2021-consert-contrastive-framework-self-supervised/)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://md.research.elamur.ai/free/en/papers/zhang-2020-unsupervised-sentence-embedding-method-mutual/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{agirre2015semeval,
  title = {{SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability}},
  author = {Agirre, Eneko and Banea, Carmen and Cardie, Claire and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei and Lopez-Gazpio, Iñigo and Maritxalar, Montse and Mihalcea, Rada and Rigau, German and Uria, Larraitz and Wiebe, Janyce},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2045},
  url = {https://doi.org/10.18653/v1/s15-2045}
}
```

- [← previous SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)](https://md.research.elamur.ai/free/en/papers/xu-2015-semeval-2015-task-1-paraphrase/)
- [next → SensEmbed: Learning Sense Embeddings for Word and Relational Similarity](https://md.research.elamur.ai/free/en/papers/iacobacci-2015-sensembed-learning-sense-embeddings-word/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Steven Abney. 1991. Parsing by chunks. In Principle-based parsing: Computation and psycholinguistics. Robert Berwick and Steven Abney and Carol Tenny(eds.), pages 257–278. [find](https://scholar.google.com/scholar?q=Steven+Abney%2E+1991%2E+Parsing+by+chunks%2E+In+Principle%2Dbased+parsing%3A+Computation+and+psycholinguistics%2E+Robert+Berwick+and+Steven+Abney+and+Carol+Tenny%28eds%2E%29%2C+pages+257%E2%80%93278%2E)
2. Rodrigo Agerri, Josu Bermudez, and German Rigau. 2014. Ixa pipeline: Efficient and ready to use multilingual NLP tools. In Proceedings of the 9th Language Resources and Evaluation Conference (LREC 2014), pages 26–31. [find](https://scholar.google.com/scholar?q=Rodrigo+Agerri%2C+Josu+Bermudez%2C+and+German+Rigau%2E+2014%2E+Ixa+pipeline%3A+Efficient+and+ready+to+use+multilingual+NLP+tools%2E+In+Proceedings+of+the+9th+Language+Resources+and+Evaluation+Conference+%28LREC+201)
3. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+and+Aitor+Gonzalez%2DAgirre%2E+2012%2E+SemEval%2D2012+Task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+%2ASEM+2012%3A+The+First+Joint+Conference+on+Lexical+and+Computational) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/)
4. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 Task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2014%2E+SemEval%2D2014+Task+10%3A+Multilingual+semantic+te) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/)
5. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2014). [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Georgiana+Dinu%2C+and+Germa%CC%81n+Kruszewski%2E+2014%2E+Don%E2%80%99t+count%2C+predict%21+A+systematic+comparison+of+context%2Dcounting+vs%2E+context%2Dpredicting+semantic+vectors%2E+In+Proceedings+of+the+52nd+Annual) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2014-don-t-count-predict-systematic/)
6. Clive Best, Erik van der Goot, Ken Blackler, Teófilo Garcia, and David Horby. 2005. Europe Media Monitor - System description. In EUR Report 22173-En, Ispra, Italy. [find](https://scholar.google.com/scholar?q=Clive+Best%2C+Erik+van+der+Goot%2C+Ken+Blackler%2C+Teo%CC%81filo+Garcia%2C+and+David+Horby%2E+2005%2E+Europe+Media+Monitor+%2D+System+description%2E+In+EUR+Report+22173%2DEn%2C+Ispra%2C+Italy%2E)
7. Chris Brockett. 2007. Aligning the RTE 2006 corpus. Microsoft Research. [find](https://scholar.google.com/scholar?q=Chris+Brockett%2E+2007%2E+Aligning+the+RTE+2006+corpus%2E+Microsoft+Research%2E)
8. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013), pages 758–764, Atlanta, Georgia, J… [find](https://scholar.google.com/scholar?q=Juri+Ganitkevitch%2C+Benjamin+Van+Durme%2C+and+Chris+Callison%2DBurch%2E+2013%2E+PPDB%3A+The+paraphrase+database%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+Computa) [our card](https://md.research.elamur.ai/free/en/papers/ganitkevitch-2013-ppdb-paraphrase-database/)
9. Omer Levy, Torsten Zesch, Ido Dagan, and Iryna Gurevych. 2013. Recognizing partial textual entailment. In ACL (2), pages 451–455. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Torsten+Zesch%2C+Ido+Dagan%2C+and+Iryna+Gurevych%2E+2013%2E+Recognizing+partial+textual+entailment%2E+In+ACL+%282%29%2C+pages+451%E2%80%93455%2E)
10. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at the International Conference on Learning Representations (ICLR 2013). [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+In+Proceedings+of+Workshop+at+the+International+Conference+on+Learning+Repr)
11. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, November. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2E+1995%2E+Wordnet%3A+A+lexical+database+for+english%2E+Commun%2E+ACM%2C+38%2811%29%3A39%E2%80%9341%2C+November%2E) [our card](https://md.research.elamur.ai/free/en/papers/lex-1994-miller-wordnet/)
12. Rodney D. Nielsen, Wayne Ward, and James H. Martin. 2009. Recognizing entailment in intelligent tutoring systems. Natural Language Engineering, 15(04):479– 501. [find](https://scholar.google.com/scholar?q=Rodney+D%2E+Nielsen%2C+Wayne+Ward%2C+and+James+H%2E+Martin%2E+2009%2E+Recognizing+entailment+in+intelligent+tutoring+systems%2E+Natural+Language+Engineering%2C+15%2804%29%3A479%E2%80%93+501%2E)
13. William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical recipes: The art of scientific computing V 2.10 with Linux or single-screen license. [find](https://scholar.google.com/scholar?q=William+H%2E+Press%2C+Saul+A%2E+Teukolsky%2C+William+T%2E+Vetterling%2C+and+Brian+P%2E+Flannery%2E+Numerical+recipes%3A+The+art+of+scientific+computing+V+2%2E10+with+Linux+or+single%2Dscreen+license%2E)
14. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT 2010, pages 139–147. [find](https://scholar.google.com/scholar?q=Cyrus+Rashtchian%2C+Peter+Young%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+2010%2E+Collecting+image+annotations+using+Amazon%E2%80%99s+Mechanical+Turk%2E+In+Proceedings+of+the+NAACL+HLT+2010+Workshop+on+Creating+Speech+) [our card](https://md.research.elamur.ai/free/en/papers/rashtchian-2010-collecting-image-annotations-amazon-s/)
15. Vasile Rus, Mihai Lintean, Cristian Moldovan, William Baggett, Nobal Niraula, and Brent Morgan. 2012. The SIMILAR corpus: A resource to foster the qualitative understanding of semantic similarity of texts. In Semantic Relations II: Enhancing Resources and Applications, The 8th Language Resources and… [find](https://scholar.google.com/scholar?q=Vasile+Rus%2C+Mihai+Lintean%2C+Cristian+Moldovan%2C+William+Baggett%2C+Nobal+Niraula%2C+and+Brent+Morgan%2E+2012%2E+The+SIMILAR+corpus%3A+A+resource+to+foster+the+qualitative+understanding+of+semantic+similarity+of+t)
16. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning-Volume 7 (CoNLL 2000), pages 127–132. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Sabine+Buchholz%2E+2000%2E+Introduction+to+the+CoNLL%2D2000+shared+task%3A+Chunking%2E+In+Proceedings+of+the+2nd+workshop+on+Learning+language+in+logic+and+the+4th+conference+on+Compu) [our card](https://md.research.elamur.ai/free/en/papers/sang-2000-introduction-conll-2000-shared-task/)
17. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. TakeLab: Systems for measuring semantic text similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), Montréal, Canada, 7-8 June. [find](https://scholar.google.com/scholar?q=Frane+S%CC%8Caric%CC%81%2C+Goran+Glavas%CC%8C%2C+Mladen+Karan%2C+Jan+S%CC%8Cnajder%2C+and+Bojana+Dalbelo+Bas%CC%8Cic%CC%81%2E+2012%2E+TakeLab%3A+Systems+for+measuring+semantic+text+similarity%2E+In+Proceedings+of+the+6th+International+Workshop+on)