---
title: "TakeLab: Systems for Measuring Semantic Text Similarity (2012)"
description: "The paper addresses the automatic scoring of how close in meaning two short texts are. 27 references, cited by 64 papers here."
authors: ["Frane Šarić", "Goran Glavaš", "Vanja Mladen Karan", "Jan Šnajder", "Bojana Dalbelo Bašić"]
year: 2012
venue: "*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)"
source: "https://aclanthology.org/S12-1060/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saric-2012-takelab-systems-measuring-semantic-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saric-2012-takelab-systems-measuring-semantic-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / TakeLab: Systems for Measuring Semantic Text Similarity

# TakeLab: Systems for Measuring Semantic Text Similarity

**Authors:** Frane Šarić, [Goran Glavaš](https://md.research.elamur.ai/free/en/authors/goran-glavas/), Vanja Mladen Karan, Jan Šnajder and Bojana Dalbelo Bašić · **Year:** 2012 · **Venue:** [\*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)](https://md.research.elamur.ai/free/en/venues/starsem/) ([\*SEM 2012](https://md.research.elamur.ai/free/en/venues/starsem/2012/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S12-1060/) · [PDF](https://aclanthology.org/S12-1060.pdf) · [OpenAlex](https://openalex.org/W1942015218)

**Citations:** 200 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/saric-2012-takelab-systems-measuring-semantic-text/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses the automatic scoring of how close in meaning two short texts are. The setting is the SemEval-2012 Semantic Text Similarity task (Task 6), where sentence pairs carry human similarity ratings on a 0–5 scale. The authors built two supervised systems, simple and syntax. Both feed many hand-designed pairwise features into support vector regression (LIBSVM). Hyperparameters are tuned by grid search within nested cross-validation, and a separate model is trained per training set. The features include:

- n-gram and skip-n-gram overlap;
- word overlap softened with WordNet similarity or weighted by information content estimated from Google Books Ngrams;
- sentence vectors summed from LSA word vectors built on NYT and Wikipedia;
- greedy lemma alignment;
- dependency-based features;
- comparisons of numbers and named entities.

Among 89 submissions, both systems landed in the top five on all three aggregate metrics, and simple ranked first by normalized Pearson. The work shows that well-engineered, largely lexical similarity signals are highly competitive. It also reveals how sensitive such models are to differences between training and test data.

## Key points
- Final rankings out of 89 systems:
  - overall Pearson: syntax 2nd, simple 3rd;
  - normalized Pearson: simple 1st, syntax 3rd;
  - weighted mean: simple 2nd, syntax 5th.
- On individual test sets, simple ranked 1st on MSRvid (0.8803) and MSRpar (0.7343). Both systems did poorly on SMTeuroparl and SMTnews; for example, simple scored 0.4771 (rank 26) and 0.3989 (rank 46).
- LSA word vectors were computed via truncated SVD of tf-idf term-document matrices, with 200 dimensions for NYT and 500 for Wikipedia. On WordSim353, the Wikipedia vectors reached a Spearman correlation of 0.62, versus 0.29–0.33 for the WordNet measures (PathLen, Lin).
- The authors attribute the SMT drop to a length mismatch. SMT training sentences averaged 30.4 tokens, while the test sets averaged only 12.3 (SMTeuroparl) and 13.5 (SMTnews). The test data also contained repeated very short identical pairs.
- During development, four features correlated strongly on every dataset: weighted word overlap, WordNet-augmented overlap, greedy lemma alignment, and vector-space similarity.

## Interesting ideas
- **Symmetric overlap scores.** Overlap is measured in both directions (how much A covers B and how much B covers A), and the two values are combined with a harmonic mean. This penalizes pairs where one text merely contains the other, which is useful for any symmetric matching score.
- **Rarity-based word weighting.** Matched words are weighted by information content derived from corpus frequencies, so shared rare words count much more than shared common ones. This is cheap to add to almost any overlap or alignment metric.
- **Greedy lemma alignment.** Lemmas are paired one-to-one in order of decreasing similarity, each pair is weighted by its rarer word, and the total is divided by the length of the longer sentence. This is a simple, fast substitute for optimal alignment that still penalizes unmatched material.
- **Dedicated numeric features.** Numbers get their own overlap features, and values that differ only in extra decimal places are treated as matches. This reflects that annotators strongly penalize mismatched figures, which is relevant for news, trade, or statistical texts.
- **Context-aware normalization.** Two adjacent tokens are merged into a compound only if the other sentence contains that compound. This handles spelling variation without a global normalization dictionary.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Zagreb](https://md.research.elamur.ai/free/en/universities/university-of-zagreb/) — University, Zagreb, Croatia · 7 works · 1,132 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Frane Šarić, Goran Glavaš, Vanja Mladen Karan, Jan Šnajder and Bojana Dalbelo Bašić. 2012. TakeLab: Systems for Measuring Semantic Text Similarity. \*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saric2012takelab,
  title = {{TakeLab: Systems for Measuring Semantic Text Similarity}},
  author = {Frane Šarić, Goran Glavaš, Vanja Mladen Karan, Jan Šnajder and Bojana Dalbelo Bašić},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  url = {https://aclanthology.org/S12-1060/}
}
```

- [← previous Re-examining Machine Translation Metrics for Paraphrase Identification](https://md.research.elamur.ai/free/en/papers/madnani-2012-re-examining-machine-translation-metrics/)
- [next → SemEval-2012 Task 2: Measuring Degrees of Relational Similarity](https://md.research.elamur.ai/free/en/papers/jurgens-2012-semeval-2012-task-2-measuring/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. [A study on similarity and relatedness using distributional and wordnet-based approaches](https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/). In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the A…
2. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. [SemEval-2012 Task 6: A pilot on semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/). In Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), in conjunction with the First Joint Conference on Lexical and Computational Seman…
3. Ramiz M. Aliguliyev. 2009. A new sentence similarity measure and sentence based extractive technique for automatic text summarization. Expert Systems with Applications, 36(4):7764–7772. [find](https://scholar.google.com/scholar?q=Ramiz+M%2E+Aliguliyev%2E+2009%2E+A+new+sentence+similarity+measure+and+sentence+based+extractive+technique+for+automatic+text+summarization%2E+Expert+Systems+with+Applications%2C+36%284%29%3A7764%E2%80%937772%2E)
4. Steven Bird. 2006. [NLTK: the natural language toolkit](https://md.research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/). In Proceedings of the COLING/ACL on Interactive presentation sessions, COLING-ACL ’06, pages 69– 72. Association for Computational Linguistics.
5. Alexander Budanitsky and Graeme Hirst. 2006. [Evaluating wordnet-based measures of lexical semantic relatedness](https://md.research.elamur.ai/free/en/papers/budanitsky-2006-evaluating-wordnet-based-measures-lexical/). Computational Linguistics, 32(1):13–47.
6. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1– 27:27. Software available at [http://www.csie](http://www.csie). ntu.edu.tw/˜cjlin/libsvm. [↗](http://www.csie)
7. Marie-Catherine De Marneffe and Christopher D. Manning. 2008. [The Stanford typed dependencies representation](https://md.research.elamur.ai/free/en/papers/marneffe-2008-stanford-typed-dependencies-representation/). In Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, pages 1–8. Association for Computational Linguistics.
8. Marie-Catherine De Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. [Generating typed dependency parses from phrase structure parses](https://md.research.elamur.ai/free/en/papers/marneffe-2006-generating-typed-dependency-parses-phrase/). In Proceedings of LREC, volume 6, pages 449–454.
9. Aminul Islam and Diana Inkpen. 2008. Semantic text similarity using corpus-based word similarity and string similarity. ACM Transactions on Knowledge Discovery from Data (TKDD), 2(2):10. [find](https://scholar.google.com/scholar?q=Aminul+Islam+and+Diana+Inkpen%2E+2008%2E+Semantic+text+similarity+using+corpus%2Dbased+word+similarity+and+string+similarity%2E+ACM+Transactions+on+Knowledge+Discovery+from+Data+%28TKDD%29%2C+2%282%29%3A10%2E)
10. Alon Lavie and Michael Denkowski. 2009. The ME-TEOR metric for automatic evaluation of machine translation. Machine translation, 23(2):105–115. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Michael+Denkowski%2E+2009%2E+The+ME%2DTEOR+metric+for+automatic+evaluation+of+machine+translation%2E+Machine+translation%2C+23%282%29%3A105%E2%80%93115%2E)
11. Claudia Leacock and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. WordNet: An electronic lexical database, 49(2):265–283. [find](https://scholar.google.com/scholar?q=Claudia+Leacock+and+Martin+Chodorow%2E+1998%2E+Combining+local+context+and+WordNet+similarity+for+word+sense+identification%2E+WordNet%3A+An+electronic+lexical+database%2C+49%282%29%3A265%E2%80%93283%2E)
12. Chin-Yew Lin and Eduard Hovy. 2003. [Automatic evaluation of summaries using n-gram co-occurrence statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/). In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 71–78. Association for Comput…
13. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the 15th international conference on Machine Learning, volume 1, pages 296–304. San Francisco. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1998%2E+An+information%2Dtheoretic+definition+of+similarity%2E+In+Proceedings+of+the+15th+international+conference+on+Machine+Learning%2C+volume+1%2C+pages+296%E2%80%93304%2E+San+Francisco%2E)
14. Ana G. Maguitman, Filippo Menczer, Heather Roinestad, and Alessandro Vespignani. 2005. Algorithmic detection of semantic similarity. In Proceedings of the 14th international conference on World Wide Web, pages 107–116. ACM. [find](https://scholar.google.com/scholar?q=Ana+G%2E+Maguitman%2C+Filippo+Menczer%2C+Heather+Roinestad%2C+and+Alessandro+Vespignani%2E+2005%2E+Algorithmic+detection+of+semantic+similarity%2E+In+Proceedings+of+the+14th+international+conference+on+World+Wide+W)
15. Jean-Baptiste Michel, Yuan Kui Shen, Aviva P. Aiden, Adrian Veres, Matthew K. Gray, et al. 2011. Quantitative analysis of culture using millions of digitized books. Science, 331(6014):176. [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Michel%2C+Yuan+Kui+Shen%2C+Aviva+P%2E+Aiden%2C+Adrian+Veres%2C+Matthew+K%2E+Gray%2C+et+al%2E+2011%2E+Quantitative+analysis+of+culture+using+millions+of+digitized+books%2E+Science%2C+331%286014%29%3A176%2E)
16. Rada Mihalcea, Courtney Corley, and Carlo Strapparava. 2006. Corpus-based and knowledge-based measures of text semantic similarity. In Proceedings of the national conference on artificial intelligence, volume 21, page 775. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea%2C+Courtney+Corley%2C+and+Carlo+Strapparava%2E+2006%2E+Corpus%2Dbased+and+knowledge%2Dbased+measures+of+text+semantic+similarity%2E+In+Proceedings+of+the+national+conference+on+artificial+intelligence)
17. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999. [find](https://scholar.google.com/scholar?q=Menlo+Park%2C+CA%3B+Cambridge%2C+MA%3B+London%3B+AAAI+Press%3B+MIT+Press%3B+1999%2E)
18. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. Proceedings of ACL-08: HLT, pages 236–244. [find](https://scholar.google.com/scholar?q=Jeff+Mitchell+and+Mirella+Lapata%2E+2008%2E+Vector%2Dbased+models+of+semantic+composition%2E+Proceedings+of+ACL%2D08%3A+HLT%2C+pages+236%E2%80%93244%2E)
19. Jesús Oliva, Jóse Ignacio Serrano, Marı́a Dolores Del Castillo, and Ángel Iglesias. 2011. SyMSS: A syntax-based measure for short-text semantic similarity. Data & Knowledge Engineering. [find](https://scholar.google.com/scholar?q=Jesu%CC%81s+Oliva%2C+Jo%CC%81se+Ignacio+Serrano%2C+Mar%C4%B1%CC%81a+Dolores+Del+Castillo%2C+and+A%CC%81ngel+Iglesias%2E+2011%2E+SyMSS%3A+A+syntax%2Dbased+measure+for+short%2Dtext+semantic+similarity%2E+Data+%26+Knowledge+Engineering%2E)
20. Eui-Kyu Park, Dong-Yul Ra, and Myung-Gil Jang. 2005. Techniques for improving web retrieval effectiveness. Information processing & management, 41(5):1207– 1223. [find](https://scholar.google.com/scholar?q=Eui%2DKyu+Park%2C+Dong%2DYul+Ra%2C+and+Myung%2DGil+Jang%2E+2005%2E+Techniques+for+improving+web+retrieval+effectiveness%2E+Information+processing+%26+management%2C+41%285%29%3A1207%E2%80%93+1223%2E)
21. Gerard Salton, Andrew Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. Communications of the ACM, 18(11):613–620. [find](https://scholar.google.com/scholar?q=Gerard+Salton%2C+Andrew+Wong%2C+and+Chung%2DShu+Yang%2E+1975%2E+A+vector+space+model+for+automatic+indexing%2E+Communications+of+the+ACM%2C+18%2811%29%3A613%E2%80%93620%2E)
22. Evan Sandhaus. 2008. The New York Times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. [find](https://scholar.google.com/scholar?q=Evan+Sandhaus%2E+2008%2E+The+New+York+Times+annotated+corpus%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+6%2812%29%3Ae26752%2E)
23. Hinrich Schütze. 1998. [Automatic word sense discrimination](https://md.research.elamur.ai/free/en/papers/schutze-1998-automatic-word-sense-discrimination/). Computational linguistics, 24(1):97–123.
24. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1–47. [find](https://scholar.google.com/scholar?q=Fabrizio+Sebastiani%2E+2002%2E+Machine+learning+in+automated+text+categorization%2E+ACM+computing+surveys+%28CSUR%29%2C+34%281%29%3A1%E2%80%9347%2E)
25. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Advances in Neural Information Processing Systems, 24. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Eric+H%2E+Huang%2C+Jeffrey+Pennington%2C+Andrew+Y%2E+Ng%2C+and+Christopher+D%2E+Manning%2E+2011%2E+Dynamic+pooling+and+unfolding+recursive+autoencoders+for+paraphrase+detection%2E+Advances+in+Neural+Inf)
26. Peter D. Turney and Patrick Pantel. 2010. [From frequency to meaning: Vector space models of semantics](https://md.research.elamur.ai/free/en/papers/turney-2010-frequency-meaning-vector-space-models/). Journal of Artificial Intelligence Research, 37(1):141– 188.
27. Stephen Wan, Mark Dras, Robert Dale, and Cécile Paris. 2006. Using dependency-based features to take the “para-farce” out of paraphrase. In Proceedings of the Australasian Language Technology Workshop, volume 2006. [find](https://scholar.google.com/scholar?q=Stephen+Wan%2C+Mark+Dras%2C+Robert+Dale%2C+and+Ce%CC%81cile+Paris%2E+2006%2E+Using+dependency%2Dbased+features+to+take+the+%E2%80%9Cpara%2Dfarce%E2%80%9D+out+of+paraphrase%2E+In+Proceedings+of+the+Australasian+Language+Technology+Worksho)

*Updated 24 October 2026*