---
title: "Evaluation methods for unsupervised word — Schnabel 2015"
description: "Schnabel and colleagues investigate how word embeddings should be judged."
authors: ["Tobias Schnabel", "Igor Labutov", "David Mimno", "Thorsten Joachims"]
year: 2015
venue: "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D15-1036/"
doi: "10.18653/v1/d15-1036"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Evaluation methods for unsupervised word embeddings

# Evaluation methods for unsupervised word embeddings

**Authors:** Tobias Schnabel, [Igor Labutov](https://md.research.elamur.ai/free/en/authors/igor-labutov/), [David Mimno](https://md.research.elamur.ai/free/en/authors/david-mimno/), [Thorsten Joachims](https://md.research.elamur.ai/free/en/authors/thorsten-joachims/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D15-1036/) · [PDF](https://aclanthology.org/D15-1036.pdf) · [DOI 10.18653/v1/d15-1036](https://doi.org/10.18653/v1/d15-1036) · [OpenAlex](https://openalex.org/W2252211741)

**Citations:** 540 citations (openalex-snapshot · 2026-10-14) · [129](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Schnabel and colleagues investigate how word embeddings should be judged. The common practice relies on borrowed benchmark inventories and single aggregate scores, which implicitly assumes there is one "best" embedding. The authors train six unsupervised models (CBOW, C&W, GloVe, Hellinger PCA, TSCCA, Sparse Random Projections) under matched conditions: a 2008 Wikipedia dump, 50 dimensions and a shared vocabulary. They then compare the models on three kinds of criteria: relatedness, neighborhood coherence and downstream performance.

The rankings of the models shift depending on the criterion. Embeddings should therefore be compared for a specific purpose, and extrinsic tasks are not a reliable proxy for general quality. The paper also introduces comparative intrinsic evaluation, in which crowd workers choose between nearest neighbors proposed by different models for a given query. These crowdsourced preferences correlate strongly with offline similarity benchmarks.

The authors release a query inventory calibrated for word frequency, part of speech and abstractness, along with all collected judgments. Finally, they show that embeddings carry substantial word-frequency information, and that this distorts cosine similarity.

## Key points
- All six models were trained on a Wikipedia snapshot from 2008-03-01, chosen to match the pretrained C&W data. The text was processed with the Stanford tokenizer, lower-cased, and had digits replaced by zeros. Every model used D = 50, and nearest-neighbor experiments used the 103,647-word intersection of the models' vocabularies.
- On 14 absolute intrinsic datasets (relatedness, analogy, categorization and selectional preference, following Baroni et al. 2014):
  - CBOW came out on top in 10 cases but did not lead on selectional preferences.
  - Random projection was the weakest model in 13 cases, with Hellinger PCA next.
  - TSCCA did disproportionately poorly on analogies.
- A logistic regression classifier separated frequent from rare words using length-normalized vectors, with thresholds from 100 to 50,000. Every embedding beat chance, and GloVe and TSCCA reached close to 100% accuracy for thresholds up to 1000.
- For C&W, a word's nearest-neighbor rank follows a power law in its corpus frequency rank, roughly nn-rank ∼ 1000 · corpus-rank^0.17. This was measured over the 1000 nearest neighbors of WordSim-353 words.
- The new query inventory and all user judgments were made public on the authors' Cornell page.

## Interesting ideas
- **Comparative evaluation instead of absolute scores:** Annotators are shown candidate neighbors from several models for the same query and simply pick the best one. This avoids having to rank unrelated pairs against each other, and choice questions are quick and cheap to crowdsource. The setup could work for comparing any retrieval or representation systems.
- **Designing query inventories deliberately:** Test words are sampled to cover a range of frequencies, parts of speech and levels of abstractness, rather than reused from older resources. This makes weak spots such as rare words visible, which matters for any benchmark built for a new language like Ukrainian.
- **Probing for nuisance signals:** A simple linear classifier checks whether the vectors encode word frequency. This kind of diagnostic can uncover unintended information in any learned representation before trusting its similarity scores.
- **Being skeptical of raw cosine similarity:** Since frequency contaminates neighbor rankings, the authors suggest treating frequency as an explicit factor in experiments and possibly learning a custom similarity metric.
- **Adapting word intrusion to embeddings:** The paper borrows the topic-model intrusion test and recasts it as a coherence check on nearest-neighbor sets. This gives a human-interpretable test for models whose dimensions themselves carry no meaning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

129 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 125 not yet.

- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://md.research.elamur.ai/free/en/papers/faruqui-2016-problems-evaluation-word-embeddings-word/)
- [Modeling Speech Acts in Asynchronous Conversations: A Neural-CRF Approach](https://md.research.elamur.ai/free/en/papers/joty-2018-modeling-speech-acts-asynchronous-conversations/)
- [Where New Words Are Born: Distributional Semantic Analysis of Neologisms and Their Semantic Neighborhoods](https://md.research.elamur.ai/free/en/papers/lex-2020-ryskina-neologisms-distributional/)
- [Prompting Instruction-tuned LLMs for Semantic Similarity Values](https://md.research.elamur.ai/free/en/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schnabel2015evaluation,
  title = {{Evaluation methods for unsupervised word embeddings}},
  author = {Schnabel, Tobias and Labutov, Igor and Mimno, David and Joachims, Thorsten},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1036},
  url = {https://doi.org/10.18653/v1/d15-1036}
}
```

- [← previous Entity-Centric Coreference Resolution with Model Stacking](https://md.research.elamur.ai/free/en/papers/clark-2015-entity-centric-coreference-resolution-model/)
- [next → Event Detection and Domain Adaptation with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-event-detection-domain-adaptation-convolutional/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas and Dan Klein. 2014. How much do word embeddings encode about syntax? In ACL: Short Papers, pages 822–827. [find](https://scholar.google.com/scholar?q=Jacob+Andreas+and+Dan+Klein%2E+2014%2E+How+much+do+word+embeddings+encode+about+syntax%3F+In+ACL%3A+Short+Papers%2C+pages+822%E2%80%93827%2E)
2. Javed Aslam and Mark Montague. 2001. Models for metasearch. In SIGIR, pages 276–284. [find](https://scholar.google.com/scholar?q=Javed+Aslam+and+Mark+Montague%2E+2001%2E+Models+for+metasearch%2E+In+SIGIR%2C+pages+276%E2%80%93284%2E)
3. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, pages 238–247. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Georgiana+Dinu%2C+and+Germa%CC%81n+Kruszewski%2E+2014%2E+Don%E2%80%99t+count%2C+predict%21+a+systematic+comparison+of+context%2Dcounting+vs%2E+context%2Dpredicting+semantic+vectors%2E+In+ACL%2C+pages+238%E2%80%93247%2E) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2014-don-t-count-predict-systematic/)
4. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. JAIR, 49:1–47. [find](https://scholar.google.com/scholar?q=Elia+Bruni%2C+Nam%2DKhanh+Tran%2C+and+Marco+Baroni%2E+2014%2E+Multimodal+distributional+semantics%2E+JAIR%2C+49%3A1%E2%80%9347%2E)
5. James McKeen Cattell. 1886. The time taken up by cerebral operations. Mind,, (42):220–242. [find](https://scholar.google.com/scholar?q=James+McKeen+Cattell%2E+1886%2E+The+time+taken+up+by+cerebral+operations%2E+Mind%2C%2C+%2842%29%3A220%E2%80%93242%2E)
6. Jonathan Chang, Jordan Boyd-Graber, Sean Gerrish, Chong Wang, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. In NIPS, pages 288–296. [find](https://scholar.google.com/scholar?q=Jonathan+Chang%2C+Jordan+Boyd%2DGraber%2C+Sean+Gerrish%2C+Chong+Wang%2C+and+David+Blei%2E+2009%2E+Reading+tea+leaves%3A+How+humans+interpret+topic+models%2E+In+NIPS%2C+pages+288%E2%80%93296%2E) [our card](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/)
7. Yanqing Chen, Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2013. The expressive power of word embeddings. arXiv preprint: 1408.3456. [find](https://scholar.google.com/scholar?q=Yanqing+Chen%2C+Bryan+Perozzi%2C+Rami+Al%2DRfou%2C+and+Steven+Skiena%2E+2013%2E+The+expressive+power+of+word+embeddings%2E+arXiv+preprint%3A+1408%2E3456%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JLMR, 12:2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+JLMR%2C+12%3A2493%E2%80%932537%2E) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
9. Paramveer S. Dhillon, Jordan Rodu, Dean P. Foster, and Lyle H. Ungar. 2012. Two step CCA: A new spectral method for estimating vector models of words. In ICML, pages 1551–1558. [find](https://scholar.google.com/scholar?q=Paramveer+S%2E+Dhillon%2C+Jordan+Rodu%2C+Dean+P%2E+Foster%2C+and+Lyle+H%2E+Ungar%2E+2012%2E+Two+step+CCA%3A+A+new+spectral+method+for+estimating+vector+models+of+words%2E+In+ICML%2C+pages+1551%E2%80%931558%2E)
10. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLIN-EAR: A library for large linear classification. JLMR, 9:1871–1874. [find](https://scholar.google.com/scholar?q=Rong%2DEn+Fan%2C+Kai%2DWei+Chang%2C+Cho%2DJui+Hsieh%2C+Xiang%2DRui+Wang%2C+and+Chih%2DJen+Lin%2E+2008%2E+LIBLIN%2DEAR%3A+A+library+for+large+linear+classification%2E+JLMR%2C+9%3A1871%E2%80%931874%2E)
11. Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at word-vectors.org. In ACL: System Demonstrations. [find](https://scholar.google.com/scholar?q=Manaal+Faruqui+and+Chris+Dyer%2E+2014%2E+Community+evaluation+and+exchange+of+word+vectors+at+word%2Dvectors%2Eorg%2E+In+ACL%3A+System+Demonstrations%2E)
12. Lev Finkelstein, Ehud Rivlin Zach Solan Gadi Wolfman Evgeniy Gabrilovich, Yossi Matias, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. TOIS, 20(1):116–131, January. [find](https://scholar.google.com/scholar?q=Lev+Finkelstein%2C+Ehud+Rivlin+Zach+Solan+Gadi+Wolfman+Evgeniy+Gabrilovich%2C+Yossi+Matias%2C+and+Eytan+Ruppin%2E+2002%2E+Placing+search+in+context%3A+The+concept+revisited%2E+TOIS%2C+20%281%29%3A116%E2%80%93131%2C+January%2E)
13. Bin Gao, Jiang Bian, and Tie-Yan Liu. 2014. WordRep: A benchmark for research on learning word representations. ICML Workshop on Knowledge-Powered Deep Learning for Text Mining. [find](https://scholar.google.com/scholar?q=Bin+Gao%2C+Jiang+Bian%2C+and+Tie%2DYan+Liu%2E+2014%2E+WordRep%3A+A+benchmark+for+research+on+learning+word+representations%2E+ICML+Workshop+on+Knowledge%2DPowered+Deep+Learning+for+Text+Mining%2E)
14. Rémi Lebret and Ronan Collobert. 2014. Word embeddings through Hellinger PCA. In EACL, pages 482–490. [find](https://scholar.google.com/scholar?q=Re%CC%81mi+Lebret+and+Ronan+Collobert%2E+2014%2E+Word+embeddings+through+Hellinger+PCA%2E+In+EACL%2C+pages+482%E2%80%93490%2E)
15. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In NIPS, pages 2177–2185. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Neural+word+embedding+as+implicit+matrix+factorization%2E+In+NIPS%2C+pages+2177%E2%80%932185%2E)
16. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015a. Improving distributional similarity with lessons learned from word embeddings. TACL. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Yoav+Goldberg%2C+and+Ido+Dagan%2E+2015a%2E+Improving+distributional+similarity+with+lessons+learned+from+word+embeddings%2E+TACL%2E)
17. Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015b. Do supervised distributional methods really learn lexical inference relations? In NAACL. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Steffen+Remus%2C+Chris+Biemann%2C+and+Ido+Dagan%2E+2015b%2E+Do+supervised+distributional+methods+really+learn+lexical+inference+relations%3F+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/)
18. Ping Li, Trevor J Hastie, and Kenneth W Church. 2006. Very sparse random projections. In KDD, pages 287–296. [find](https://scholar.google.com/scholar?q=Ping+Li%2C+Trevor+J+Hastie%2C+and+Kenneth+W+Church%2E+2006%2E+Very+sparse+random+projections%2E+In+KDD%2C+pages+287%E2%80%93296%2E)
19. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In HLT-ACL, pages 142–150. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+HLT%2DACL%2C+pages+142%E2%80%93150%2E) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
20. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL: System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+ACL%3A+System+Demonstrations%2C) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013a%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+NIPS%2C+pages+3111%E2%80%933119%2E)
22. Tomas Mikolov, Wen-Tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746– 751. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Wen%2DTau+Yih%2C+and+Geoffrey+Zweig%2E+2013b%2E+Linguistic+regularities+in+continuous+space+word+representations%2E+In+HLT%2DNAACL%2C+pages+746%E2%80%93+751%2E) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/)
23. George A Miller. 1995. WordNet: a lexical database for english. Communications of the ACM, 38(11):39–41. [find](https://scholar.google.com/scholar?q=George+A+Miller%2E+1995%2E+WordNet%3A+a+lexical+database+for+english%2E+Communications+of+the+ACM%2C+38%2811%29%3A39%E2%80%9341%2E) [our card](https://md.research.elamur.ai/free/en/papers/lex-1994-miller-wordnet/)
24. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In HLT-NAACL, pages 100–108. [find](https://scholar.google.com/scholar?q=David+Newman%2C+Jey+Han+Lau%2C+Karl+Grieser%2C+and+Timothy+Baldwin%2E+2010%2E+Automatic+evaluation+of+topic+coherence%2E+In+HLT%2DNAACL%2C+pages+100%E2%80%93108%2E) [our card](https://md.research.elamur.ai/free/en/papers/newman-2010-automatic-evaluation-topic-coherence/)
25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2E)
26. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In EMNLP. [find](https://scholar.google.com/scholar?q=Yulia+Tsvetkov%2C+Manaal+Faruqui%2C+Wang+Ling%2C+Guillaume+Lample%2C+and+Chris+Dyer%2E+2015%2E+Evaluation+of+word+vector+representations+by+subspace+alignment%2E+In+EMNLP%2E)
27. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL, pages 384– 394. [find](https://scholar.google.com/scholar?q=Joseph+Turian%2C+Lev+Ratinov%2C+and+Yoshua+Bengio%2E+2010%2E+Word+representations%3A+a+simple+and+general+method+for+semi%2Dsupervised+learning%2E+In+ACL%2C+pages+384%E2%80%93+394%2E) [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
28. Peter D. Turney. 2008. A uniform approach to analogies, synonyms, antonyms, and associations. In COLING, pages 905–912. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney%2E+2008%2E+A+uniform+approach+to+analogies%2C+synonyms%2C+antonyms%2C+and+associations%2E+In+COLING%2C+pages+905%E2%80%93912%2E)
29. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In ACL, pages 947–953. [find](https://scholar.google.com/scholar?q=Alexander+Yeh%2E+2000%2E+More+accurate+tests+for+the+statistical+significance+of+result+differences%2E+In+ACL%2C+pages+947%E2%80%93953%2E)