---
title: "Analogy-based detection of morphological and — Gladkova 2016"
description: "Analogy tests in the style of king − man + woman = queen are widely used as evidence that word embeddings encode linguistic relations."
authors: ["Anna Gladkova", "Aleksandr Drozd", "Satoshi Matsuoka"]
year: 2016
venue: "Proceedings of the NAACL Student Research Workshop"
source: "https://aclanthology.org/N16-2002/"
doi: "10.18653/v1/n16-2002"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gladkova-2016-analogy-based-detection-morphological-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gladkova-2016-analogy-based-detection-morphological-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't.

# Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't.

**Authors:** Anna Gladkova, Aleksandr Drozd, Satoshi Matsuoka · **Year:** 2016 · **Venue:** Proceedings of the NAACL Student Research Workshop · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N16-2002/) · [PDF](https://aclanthology.org/N16-2002.pdf) · [DOI 10.18653/v1/n16-2002](https://doi.org/10.18653/v1/n16-2002) · [OpenAlex](https://openalex.org/W2460442863)

**Citations:** 202 citations (openalex-snapshot · 2026-10-14) · [109](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Analogy tests in the style of king − man + woman = queen are widely used as evidence that word embeddings encode linguistic relations. However, the popular Google analogy set covers few relation types and is dominated by a handful of them. The authors built BATS (Bigger Analogy Test Set), a balanced benchmark of 40 relations split evenly across four groups: inflectional morphology, derivational morphology, lexicographic semantics and encyclopedic semantics. Using the vector offset method, they evaluated a count-based model (PMI weighting followed by SVD) and GloVe, and tracked how window size and vector dimensionality affect each relation separately rather than only the overall average.

Accuracy turned out to be modest: 22.1% for the best SVD configuration and 28.5% for GloVe. Derivational and lexicographic relations were especially hard. Both models found the same categories easy or difficult, which supports the view that count-based and neural embeddings are conceptually similar. The best parameter settings depended on the individual relation, not on whether it was semantic or morphological. This suggests that earlier analogy results overstated how well embeddings capture linguistic relations in general.

## Key points
- BATS contains 40 categories with 50 unique word pairs each, which gives 2,480 questions per category and 99,200 in total. The set ships with a Python evaluation script, available at vsm.blackbird.pw/bats.
- Sources for the categories:
  - The lexicographic part extends SemEval2012-Task2, with about 15% of the added words taken from BLESS and EVALution.
  - The encyclopedic part draws on Wikipedia lists and other web resources.
  - E01 and E10 come from the Google test set, and E09 comes from the color dataset of Bruni et al.
- Best average accuracies were 22.1% for SVD (window 3, 1000 dimensions) and 28.5% for GloVe (window 10, 300 dimensions).
- For the SVD model, windows of 2–4 worked best for both semantic and morphological relations. This contradicts the common belief that wider windows help semantics and narrower ones help morphology.
- Higher dimensionality affected categories differently:
  - 19 categories (47.5%) improved up to or kept improving at 1200 dimensions.
  - 40% peaked somewhere between 200 and 1100 dimensions, or declined as dimensions grew.
  - In pilot runs, changing the exponent applied to the SVD Σ matrix shifted per-category accuracy by 40–50%.

## Interesting ideas
- **Control homonymy when sampling test items.** The authors kept only words that WordNet assigns to a single part of speech, then picked the most frequent remaining pairs. Filtering by POS ambiguity before frequency would make morphology benchmarks for other languages, including Ukrainian, measure the intended relation more cleanly.
- **Accept several correct answers.** Each question allows multiple valid answers drawn from WordNet (for example, two different hypernyms of the same word). This avoids penalizing a model for a correct answer that the gold list happens to omit, which matters for any lexical benchmark.
- **Balance relation types and always report per-category results.** Averages can hide very large gaps between relations. In earlier work on the Google set, one relation type made up over half of the semantic questions. A balanced design with per-category breakdowns gives a more honest picture of what a model can and cannot do.
- **Tune hyperparameters for the target relation.** Since no window size or dimensionality was best across categories, an application that relies on one specific relation should be tuned and validated on that relation rather than on a general benchmark.
- **Weight SVD singular values with an exponent.** Raising Σ to a power below 1 gives more weight to low-variance dimensions, which may carry subtler linguistic distinctions. This is a cheap setting that can change results substantially in any count-based embedding pipeline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

109 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 107 not yet.

- [A Distributional and Orthographic Aggregation Model for English Derivational Morphology](https://md.research.elamur.ai/free/en/papers/deutsch-2018-distributional-orthographic-aggregation-model-english/)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gladkova2016analogy,
  title = {{Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't.}},
  author = {Gladkova, Anna and Drozd, Aleksandr and Matsuoka, Satoshi},
  booktitle = {Proceedings of the NAACL Student Research Workshop},
  year = {2016},
  doi = {10.18653/v1/n16-2002},
  url = {https://doi.org/10.18653/v1/n16-2002}
}
```

- [← previous An Evaluation of Parser Robustness for Ungrammatical Sentences](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-hashemi-evaluation-parser-robustness-ungrammatical-sentences/)
- [next → Analyzing lexical emergence in Modern American English online](https://md.research.elamur.ai/free/en/papers/lex-2016-grieve-lexical-emergence-american-english/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marco Baroni and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, GEMS ’11, pages 1–10. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marco+Baroni+and+Alessandro+Lenci%2E+2011%2E+How+we+BLESSed+distributional+semantic+evaluation%2E+In+Proceedings+of+the+GEMS+2011+Workshop+on+GEometrical+Models+of+Natural+Language+Semantics%2C+GEMS+%E2%80%9911%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/)
2. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Silvia+Bernardini%2C+Adriano+Ferraresi%2C+and+Eros+Zanchetta%2E+2009%2E+The+wacky+wide+web%3A+a+collection+of+very+large+linguistically+processed+web%2Dcrawled+corpora%2E+Language+Resources+and+Evalua)
3. Marco Baroni, Georgiana Dinu, and Germn Kruszewski. 2014. Dont count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 238–247. Vladimı́r Benko. 20… [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Georgiana+Dinu%2C+and+Germn+Kruszewski%2E+2014%2E+Dont+count%2C+predict%21+a+systematic+comparison+of+context%2Dcounting+vs%2E+context%2Dpredicting+semantic+vectors%2E+In+Proceedings+of+the+52nd+Annual+Me) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2014-don-t-count-predict-systematic/)
4. Springer. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136–145. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Springer%2E+Elia+Bruni%2C+Gemma+Boleda%2C+Marco+Baroni%2C+and+Nam%2DKhanh+Tran%2E+2012%2E+Distributional+semantics+in+technicolor%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Ling) [our card](https://md.research.elamur.ai/free/en/papers/bruni-2012-distributional-semantics-technicolor/)
5. John A. Bullinaria and Joseph P. Levy. 2012. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD. Behavior Research Methods, 44(3):890–907. [find](https://scholar.google.com/scholar?q=John+A%2E+Bullinaria+and+Joseph+P%2E+Levy%2E+2012%2E+Extracting+semantic+representations+from+word+co%2Doccurrence+statistics%3A+stop%2Dlists%2C+stemming%2C+and+SVD%2E+Behavior+Research+Methods%2C+44%283%29%3A890%E2%80%93907%2E)
6. John Caron. 2001. Computational information retrieval. chapter Experiments with LSA Scoring: Optimal Rank and Basis, pages 157–169. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. [find](https://scholar.google.com/scholar?q=John+Caron%2E+2001%2E+Computational+information+retrieval%2E+chapter+Experiments+with+LSA+Scoring%3A+Optimal+Rank+and+Basis%2C+pages+157%E2%80%93169%2E+Society+for+Industrial+and+Applied+Mathematics%2C+Philadelphia%2C+PA%2C+US)
7. Qing Cui, Bin Gao, Jiang Bian, Siyu Qiu, and Tie-Yan Liu. 2014. Learning effective word embedding using morphological word similarity. [find](https://scholar.google.com/scholar?q=Qing+Cui%2C+Bin+Gao%2C+Jiang+Bian%2C+Siyu+Qiu%2C+and+Tie%2DYan+Liu%2E+2014%2E+Learning+effective+word+embedding+using+morphological+word+similarity%2E)
8. Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2015. Python, performance, and natural language processing. In Proceedings of the 5th Workshop on Python for High-Performance and Scientific Computing, PyHPC ’15, pages 1:1–1:10, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Aleksandr+Drozd%2C+Anna+Gladkova%2C+and+Satoshi+Matsuoka%2E+2015%2E+Python%2C+performance%2C+and+natural+language+processing%2E+In+Proceedings+of+the+5th+Workshop+on+Python+for+High%2DPerformance+and+Scientific+Compu)
9. ACM. Stefano Federici, Simonetta Montemagni, and Vito Pirrelli. 1997. Inferring semantic similarity from distributional evidence: an analogy-based approach to word sense disambiguation. In Proceedings of the ACL/EACL Workshop on Automatic Information Extraction and Building of Lexical Semantic Resou… [find](https://scholar.google.com/scholar?q=ACM%2E+Stefano+Federici%2C+Simonetta+Montemagni%2C+and+Vito+Pirrelli%2E+1997%2E+Inferring+semantic+similarity+from+distributional+evidence%3A+an+analogy%2Dbased+approach+to+word+sense+disambiguation%2E+In+Proceedings)
10. Daniel Fried and Kevin Duh. 2015. Incorporating both distributional and relational semantics in word representations. [find](https://scholar.google.com/scholar?q=Daniel+Fried+and+Kevin+Duh%2E+2015%2E+Incorporating+both+distributional+and+relational+semantics+in+word+representations%2E)
11. Ama Herdadelen and Marco Baroni. 2009. BagPack: A general framework to represent semantic relations. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, GEMS ’09, pages 33– 40. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ama+Herdadelen+and+Marco+Baroni%2E+2009%2E+BagPack%3A+A+general+framework+to+represent+semantic+relations%2E+In+Proceedings+of+the+Workshop+on+Geometrical+Models+of+Natural+Language+Semantics%2C+GEMS+%E2%80%9909%2C+pages)
12. David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and… [find](https://scholar.google.com/scholar?q=David+A%2E+Jurgens%2C+Peter+D%2E+Turney%2C+Saif+M%2E+Mohammad%2C+and+Keith+J%2E+Holyoak%2E+2012%2E+Semeval%2D2012+task+2%3A+Measuring+degrees+of+relational+similarity%2E+In+Proceedings+of+the+First+Joint+Conference+on+Lexica)
13. Douwe Kiela and Stephen Clark. 2014. A systematic study of semantic vector space model parameters. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC) at EACL, pages 21–30. Maximilian Köper, Christian Scheible, and Sabine Schulte im Walde. 2015. Mu… [find](https://scholar.google.com/scholar?q=Douwe+Kiela+and+Stephen+Clark%2E+2014%2E+A+systematic+study+of+semantic+vector+space+model+parameters%2E+In+Proceedings+of+the+2nd+Workshop+on+Continuous+Vector+Space+Models+and+their+Compositionality+%28CVSC)
14. Siwei Lai, Kang Liu, Liheng Xu, and Jun Zhao. 2015. How to generate a good word embedding? Gabriella Lapesa and Stefan Evert. 2014. A large scale evaluation of distributional semantic models: Parameters, interactions and model selection. 2:531–545. Jean-Franois Lavallée and Philippe Langlais. 2010.… [find](https://scholar.google.com/scholar?q=Siwei+Lai%2C+Kang+Liu%2C+Liheng+Xu%2C+and+Jun+Zhao%2E+2015%2E+How+to+generate+a+good+word+embedding%3F+Gabriella+Lapesa+and+Stefan+Evert%2E+2014%2E+A+large+scale+evaluation+of+distributional+semantic+models%3A+Paramete)
15. Springer. Rmi Lebret and Ronan Collobert. 2015. Rehabilitation of count-based models for word vector representations. In Computational Linguistics and Intelligent Text Processing, pages 417–429. [find](https://scholar.google.com/scholar?q=Springer%2E+Rmi+Lebret+and+Ronan+Collobert%2E+2015%2E+Rehabilitation+of+count%2Dbased+models+for+word+vector+representations%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+417%E2%80%93429%2E)
16. Springer. Yves Lepage and Chooi-ling Goh. 2009. Towards automatic acquisition of linguistic features. In Proceedings of the 17th Nordic Conference on Computational Linguistics (NODALIDA 2009), eds., Kristiina Jokinen and Eckard Bick, pages 118–125. [find](https://scholar.google.com/scholar?q=Springer%2E+Yves+Lepage+and+Chooi%2Dling+Goh%2E+2009%2E+Towards+automatic+acquisition+of+linguistic+features%2E+In+Proceedings+of+the+17th+Nordic+Conference+on+Computational+Linguistics+%28NODALIDA+2009%29%2C+eds%2E%2C+K)
17. Omer Levy and Yoav Goldberg. 2014a. Dependency-based word embeddings. In ACL (2), pages 302–308. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014a%2E+Dependency%2Dbased+word+embeddings%2E+In+ACL+%282%29%2C+pages+302%E2%80%93308%2E)
18. Omer Levy and Yoav Goldberg. 2014b. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Ann Arbor, Michigan, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014b%2E+Linguistic+regularities+in+sparse+and+explicit+word+representations%2E+In+Proceedings+of+the+Eighteenth+Conference+on+Computational+Natural+Language+Learning%2C+pages+1) [our card](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)
19. Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. 2014. Linguistic regularities in sparse and explicit word representations. In CoNLL, pages 171–180. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Yoav+Goldberg%2C+and+Israel+Ramat%2DGan%2E+2014%2E+Linguistic+regularities+in+sparse+and+explicit+word+representations%2E+In+CoNLL%2C+pages+171%E2%80%93180%2E) [our card](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)
20. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. In Transactions of the Association for Computational Linguistics, volume 3, pages 211–225. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Yoav+Goldberg%2C+and+Ido+Dagan%2E+2015%2E+Improving+distributional+similarity+with+lessons+learned+from+word+embeddings%2E+In+Transactions+of+the+Association+for+Computational+Linguistics%2C+volume+3)
21. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North Am… [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013a%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+Tomas+Mikolov%2C+Wen+tau+Yih%2C+and+Geoffrey+Zweig%2E+2013b%2E+Linguistic+regulari)
22. George Miller and Christiane Fellbaum. 1998. Wordnet: An electronic lexical database. [find](https://scholar.google.com/scholar?q=George+Miller+and+Christiane+Fellbaum%2E+1998%2E+Wordnet%3A+An+electronic+lexical+database%2E)
23. MIT Press: Cambridge. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), volume 12, pages 1532–1543. [find](https://scholar.google.com/scholar?q=MIT+Press%3A+Cambridge%2E+Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+Empiricial+Methods+in+Natural+Language+)
24. Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In Proceedings of the 4th Workshop on Linked Data in Linguistics (LDL-2015), pages 64–69. [find](https://scholar.google.com/scholar?q=Enrico+Santus%2C+Frances+Yung%2C+Alessandro+Lenci%2C+and+Chu%2DRen+Huang%2E+2015%2E+EVALution+1%2E0%3A+an+evolving+semantic+dataset+for+training+and+evaluation+of+distributional+semantic+models%2E+In+Proceedings+of+the)
25. Radu Soricut and Franz Och. 2015. Unsupervised morphology induction using word embeddings. In Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1627–1637. [find](https://scholar.google.com/scholar?q=Radu+Soricut+and+Franz+Och%2E+2015%2E+Unsupervised+morphology+induction+using+word+embeddings%2E+In+Human+Language+Technologies%3A+The+2015+Annual+Conference+of+the+North+American+Chapter+of+the+ACL%2C+pages+16)
26. Peter Turney, Michael Littman, Jeffrey Bigham, and Victor Shnayder. 2003. Combining independent modules to solve multiple-choice synonym and analogy problems. In In Proceedings of the International Conference on Recent Advances in Natural Language Processing, pages 482–489. [find](https://scholar.google.com/scholar?q=Peter+Turney%2C+Michael+Littman%2C+Jeffrey+Bigham%2C+and+Victor+Shnayder%2E+2003%2E+Combining+independent+modules+to+solve+multiple%2Dchoice+synonym+and+analogy+problems%2E+In+In+Proceedings+of+the+International+Co)
27. Peter D. Turney. 2008. A uniform approach to analogies, synonyms, antonyms, and associations. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 905–912. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney%2E+2008%2E+A+uniform+approach+to+analogies%2C+synonyms%2C+antonyms%2C+and+associations%2E+In+Proceedings+of+the+22nd+International+Conference+on+Computational+Linguistics+%28Coling+2008%29%2C+pages+905%E2%80%93)
28. Ekaterina Vylomova, Laura Rimmel, Trevor Cohn, and Timothy Baldwin. 2015. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. [find](https://scholar.google.com/scholar?q=Ekaterina+Vylomova%2C+Laura+Rimmel%2C+Trevor+Cohn%2C+and+Timothy+Baldwin%2E+2015%2E+Take+and+took%2C+gaggle+and+goose%2C+book+and+read%3A+Evaluating+the+utility+of+vector+differences+for+lexical+relation+learning%2E)
29. Christian Wartena. 2014. On the effect of word frequency on distributional similarity. In Proceedings of the 12th edition of the KONVENS conference -Hildesheim, volume 1, pages 1–10. [find](https://scholar.google.com/scholar?q=Christian+Wartena%2E+2014%2E+On+the+effect+of+word+frequency+on+distributional+similarity%2E+In+Proceedings+of+the+12th+edition+of+the+KONVENS+conference+%2DHildesheim%2C+volume+1%2C+pages+1%E2%80%9310%2E)