--- title: "SemEval-2012 Task 6 — Agirre 2012" description: "The paper describes the pilot SemEval-2012 Task 6 shared task on Semantic Textual Similarity (STS)." authors: ["Eneko Agirre", "Daniel Cer", "Mona Diab", "Aitor Gonzalez-Agirre"] year: 2012 venue: "*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)" source: "https://aclanthology.org/S12-1051/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/) / SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity # SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity **Authors:** [Eneko Agirre](https://md.research.elamur.ai/free/en/authors/eneko-agirre/), [Daniel Cer](https://md.research.elamur.ai/free/en/authors/daniel-cer/), [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/), [Aitor Gonzalez-Agirre](https://md.research.elamur.ai/free/en/authors/aitor-gonzalez-agirre/) · **Year:** 2012 · **Venue:** \*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/S12-1051/) · [PDF](https://aclanthology.org/S12-1051.pdf) · [OpenAlex](https://openalex.org/W2251861449) **Citations:** 645 citations (openalex-snapshot · 2026-10-14) · [462](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper describes the pilot SemEval-2012 Task 6 shared task on Semantic Textual Similarity (STS). Participants had to judge how close in meaning two sentences are. Unlike textual entailment and paraphrase, similarity here is symmetric and graded rather than a binary "yes/no". No comparable dataset for this task existed before, so the organizers assembled sentence pairs from existing resources. For training, they took 2000 pairs from MSRpar, MSRvid and "reference translation – machine translation" pairs from WMT. The test set contains another 2000 pairs from the same sources plus two surprise out-of-domain datasets. Pairs were selected by bands of string similarity in order to cover different levels of similarity evenly. Each pair was rated by crowdsourced annotators on Amazon Mechanical Turk on a 0–5 scale, where each score has a verbal definition. Systems were compared with human judgments using Pearson correlation. The best systems exceeded 80%, whereas a simple lexical baseline scored only 31%. This paper laid the foundation for subsequent STS benchmarks. At the same time, the authors acknowledge that the evaluation metric still needs refinement. ## Key points - **Data:** 2000 training and 2000 test pairs from MSRpar, MSRvid and SMTeuroparl (WMT 2007/2008, fr-en only). Surprise test sets: about 400 pairs from WMT 2007 news (399 in practice, due to duplicates) and 750 pairs of definitions (glosses) from OntoNotes 4.0 and WordNet 3.1. - **Annotation:** Amazon Mechanical Turk, 5 ratings for each HIT of 5 pairs, paid $0.20 per HIT. Annotators were required to have a 95% approval rating and to pass a qualification test. Ratings from annotators whose correlation with the gold standard was below 50% were discarded. - **Agreement:** in the authors' own pilot annotation, the Pearson correlation between them was 84–87%. AMT annotator agreement was about 90%. - **Participation and results:** 35 teams submitted 88 runs. The best systems achieved a Pearson correlation above 0.80, the lexical baseline 0.31. - **Open questions:** in the end, three different evaluation metrics were published, since none proved fully satisfactory. Among the resources most often used by participants were WordNet, Wikipedia, monolingual corpora, as well as machine learning and string similarity tools. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cited by Cited by 462 papers of our library cite it (of the 140884 whose bibliographies we read); 29 of them have a page here, 433 not yet. - [Explanations for CommonsenseQA: New Dataset and Models](https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/) - [*SEM 2013 shared task: Semantic Textual Similarity](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/) - [SemEval-2014 Task 10: Multilingual Semantic Textual Similarity](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/) - [SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability](https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/) - [SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/) - [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/) - [What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/) - [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/) - [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/) - [Effective Parallel Corpus Mining using Bilingual Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/guo-2018-effective-parallel-corpus-mining-bilingual/) - [UMBC_EBIQUITY-CORE: Semantic Textual Similarity Systems](https://md.research.elamur.ai/free/en/papers/han-2013-umbc-ebiquity-core-semantic-textual/) - [An Analysis of Natural Language Inference Benchmarks through the Lens of Negation](https://md.research.elamur.ai/free/en/papers/hossain-2020-analysis-natural-language-inference-benchmarks/) - [ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation](https://md.research.elamur.ai/free/en/papers/huang-2023-paraamr-large-scale-syntactically-diverse/) - [Self-Guided Contrastive Learning for BERT Sentence Representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/) - [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://md.research.elamur.ai/free/en/papers/lee-2024-nv-embed-improved-techniques-training/) - [On the Sentence Embeddings from Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/) - [SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment](https://md.research.elamur.ai/free/en/papers/marelli-2014-semeval-2014-task-1-evaluation/) - [Learning To Use Formulas To Solve Simple Arithmetic Problems](https://md.research.elamur.ai/free/en/papers/mitra-2016-learning-use-formulas-solve-simple/) - [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/) - [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/) - [Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/) - [A Pragmatics-Centered Evaluation Framework for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/sileo-2022-pragmatics-centered-evaluation-framework-natural/) - [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://md.research.elamur.ai/free/en/papers/su-2022-one-embedder-any-task-instruction/) - [Towards Universal Paraphrastic Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/) - [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/) - [C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/) - [SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)](https://md.research.elamur.ai/free/en/papers/xu-2015-semeval-2015-task-1-paraphrase/) - [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://md.research.elamur.ai/free/en/papers/yan-2021-consert-contrastive-framework-self-supervised/) - [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://md.research.elamur.ai/free/en/papers/zhang-2020-unsupervised-sentence-embedding-method-mutual/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{agirre2012semeval, title = {{SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity}}, author = {Agirre, Eneko and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor}, booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)}, year = {2012}, url = {https://aclanthology.org/S12-1051/} } ``` - [← previous SemEval-2012 Task 1: English Lexical Simplification](https://md.research.elamur.ai/free/en/papers/specia-2012-semeval-2012-task-1-english/) - [next → SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning](https://md.research.elamur.ai/free/en/papers/gordon-2012-semeval-2012-task-7-choice/) - [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/) ## References 10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 136–158. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Cameron+Fordyce%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2007%2E+%28meta%2D%29+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+T) [our card](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/) 2. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, StatMT ’08, pages 70–106. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Cameron+Fordyce%2C+Philipp+Koehn%2C+Christof+Monz%2C+and+Josh+Schroeder%2E+2008%2E+Further+meta%2Devaluation+of+machine+translation%2E+In+Proceedings+of+the+Third+Workshop+on+Statistical+Machi) 3. David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meetings of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=David+L%2E+Chen+and+William+B%2E+Dolan%2E+2011%2E+Collecting+highly+parallel+data+for+paraphrase+evaluation%2E+In+Proceedings+of+the+49th+Annual+Meetings+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/) 4. B. Dolan, C. Quirk, and C. Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COLING 04: Proceedings of the 20th international conference on Computational Linguistics, page 350. [find](https://scholar.google.com/scholar?q=B%2E+Dolan%2C+C%2E+Quirk%2C+and+C%2E+Brockett%2E+2004%2E+Unsupervised+construction+of+large+paraphrase+corpora%3A+Exploiting+massively+parallel+news+sources%2E+In+COLING+04%3A+Proceedings+of+the+20th+international+confer) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2004-unsupervised-construction-large-paraphrase-corpora/) 5. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+MIT+Press%2E) 6. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2006%2E+Ontonotes%3A+The+90%25+solution%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Ch) 7. Michael D. Lee, Brandon Pincombe, and Matthew Welsh. 2005. An empirical evaluation of models of text document similarity. In Proceedings of the 27th Annual Conference of the Cognitive Science Society, pages 1254–1259, Mahwah, NJ. [find](https://scholar.google.com/scholar?q=Michael+D%2E+Lee%2C+Brandon+Pincombe%2C+and+Matthew+Welsh%2E+2005%2E+An+empirical+evaluation+of+models+of+text+document+similarity%2E+In+Proceedings+of+the+27th+Annual+Conference+of+the+Cognitive+Science+Society%2C) 8. Y. Li, D. McLean, Z. A. Bandar, J. D. O’Shea, and K. Crockett. 2006. Sentence similarity based on semantic nets and corpus statistics. IEEE Transactions on Knowledge and Data Engineering, 18(8):1138– 1150, August. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+D%2E+McLean%2C+Z%2E+A%2E+Bandar%2C+J%2E+D%2E+O%E2%80%99Shea%2C+and+K%2E+Crockett%2E+2006%2E+Sentence+similarity+based+on+semantic+nets+and+corpus+statistics%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+18%288%29%3A1138%E2%80%93+) 9. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM, 8(10):627–633, October. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+B%2E+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Commun%2E+ACM%2C+8%2810%29%3A627%E2%80%93633%2C+October%2E) 10. E. Ukkonen. 1985. Algorithms for approximate string matching. Information and Contro, 64:110–118. [find](https://scholar.google.com/scholar?q=E%2E+Ukkonen%2E+1985%2E+Algorithms+for+approximate+string+matching%2E+Information+and+Contro%2C+64%3A110%E2%80%93118%2E)