---
title: "Do Supervised Distributional Methods Really — Levy 2015"
description: "This paper asks whether supervised classifiers built on distributional word vectors actually learn lexical inference relations, such as hypernymy and…"
authors: ["Omer Levy", "Steffen Remus", "Chris Biemann", "Ido Dagan"]
year: 2015
venue: "Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N15-1098/"
doi: "10.3115/v1/n15-1098"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Do Supervised Distributional Methods Really Learn Lexical Inference Relations?

# Do Supervised Distributional Methods Really Learn Lexical Inference Relations?

**Authors:** [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/), Steffen Remus, [Chris Biemann](https://md.research.elamur.ai/free/en/authors/chris-biemann/), [Ido Dagan](https://md.research.elamur.ai/free/en/authors/ido-dagan/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N15-1098/) · [PDF](https://aclanthology.org/N15-1098.pdf) · [DOI 10.3115/v1/n15-1098](https://doi.org/10.3115/v1/n15-1098) · [OpenAlex](https://openalex.org/W2296076036)

**Citations:** 207 citations (openalex-snapshot · 2026-10-14) · [118](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This paper asks whether supervised classifiers built on distributional word vectors actually learn lexical inference relations, such as hypernymy and entailment, between two words x and y. The authors built 9 word representations from Wikipedia (1.5 billion tokens) by combining three context types (bag-of-words, positional, dependency) with three models (PPMI, SVD, SGNS). They tested the pair compositions concat, diff, only x and only y using logistic regression and SVM classifiers on five labeled datasets. They found that much of the reported success comes from lexical memorization. When train and test vocabularies are kept apart, the classifiers mostly judge whether y looks like a typical category word, a "prototypical hypernym", and largely ignore x. A kernel analysis explains why: linear and polynomial kernels over concat or diff have no term for the similarity between x and y. Adding that similarity through new kernels helps only slightly. The authors conclude that contextual features of individual words may not contain relational information at all, which matters for anyone designing entailment or hypernymy classifiers.

## Key points
- **Datasets:** Kotlerman 2010, BLESS 2011, Baroni 2012, Turney 2014 and Levy 2014. The authors released splits with zero lexical overlap between train and test for replication.
- **Lexical memorization:** A classifier using only a one-hot encoding of y comes within 10 F1 points of contextual-feature classifiers on the larger datasets.
- **Weak baselines beat or match supervision:**
  - Unsupervised cosine similarity outperforms every supervised method on 2 of 5 datasets.
  - The only-y composition is within 5 F1 points of the best configuration on 4 of 5 datasets.
- **Mismatched pairs test:** On synthetic pairs like (banana, animal), the false-positive rate tracks recall almost exactly (match error ≈ 0.935 · recall). Top-weighted features include Hearst-pattern cues such as "such", "other" and "including".
- **Kernel fixes:** An RBF kernel over diff and a new SIM kernel that includes x·y reduce match error to ≈ 0.618 · recall. However, F1 improves only marginally.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

118 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 113 not yet.

- [SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
- [Lexical Features in Coreference Resolution: To be Used With Caution](https://md.research.elamur.ai/free/en/papers/moosavi-2017-lexical-features-coreference-resolution-be/)
- [Integrating Predictions from Neural-Network Relation Classifiers into Coreference and Bridging Resolution](https://md.research.elamur.ai/free/en/papers/roesiger-2018-integrating-predictions-neural-network-relation/)
- [Evaluation methods for unsupervised word embeddings](https://md.research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://md.research.elamur.ai/free/en/papers/shwartz-2016-improving-hypernymy-detection-integrated-path/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{levy2015do,
  title = {{Do Supervised Distributional Methods Really Learn Lexical Inference Relations?}},
  author = {Levy, Omer and Remus, Steffen and Biemann, Chris and Dagan, Ido},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1098},
  url = {https://doi.org/10.3115/v1/n15-1098}
}
```

- [← previous Distilling the Knowledge in a Neural Network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
- [next → Document Modeling with Gated Recurrent Neural Network for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673– 721. [find](https://scholar.google.com/scholar?q=Marco+Baroni+and+Alessandro+Lenci%2E+2010%2E+Distributional+memory%3A+A+general+framework+for+corpus%2Dbased+semantics%2E+Computational+Linguistics%2C+36%284%29%3A673%E2%80%93+721%2E) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2010-distributional-memory-general-framework-corpus/)
2. Marco Baroni and Alessandro Lenci. 2011. How we blessed distributional semantic evaluation. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 1–10, Edinburgh, UK. [find](https://scholar.google.com/scholar?q=Marco+Baroni+and+Alessandro+Lenci%2E+2011%2E+How+we+blessed+distributional+semantic+evaluation%2E+In+Proceedings+of+the+GEMS+2011+Workshop+on+GEometrical+Models+of+Natural+Language+Semantics%2C+pages+1%E2%80%9310%2C+Ed) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/)
3. Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 23–32, Avignon, France. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Raffaella+Bernardi%2C+Ngoc%2DQuynh+Do%2C+and+Chung%2Dchieh+Shan%2E+2012%2E+Entailment+above+the+word+level+in+distributional+semantics%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+) [our card](https://md.research.elamur.ai/free/en/papers/baroni-2012-entailment-above-word-level-distributional/)
4. Chris Biemann and Martin Riedl. 2013. Text: Now in 2D! A framework for lexical expansion with contextual similarity. Journal of Language Modelling, 1(1):55–95. [find](https://scholar.google.com/scholar?q=Chris+Biemann+and+Martin+Riedl%2E+2013%2E+Text%3A+Now+in+2D%21+A+framework+for+lexical+expansion+with+contextual+similarity%2E+Journal+of+Language+Modelling%2C+1%281%29%3A55%E2%80%9395%2E) [our card](https://md.research.elamur.ai/free/en/papers/biemann-2013-text-now-2d-framework-lexical/)
5. John Caron. 2001. Experiments with LSA scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157–169. [find](https://scholar.google.com/scholar?q=John+Caron%2E+2001%2E+Experiments+with+LSA+scoring%3A+Optimal+rank+and+basis%2E+In+Proceedings+of+the+SIAM+Computational+Information+Retrieval+Workshop%2C+pages+157%E2%80%93169%2E)
6. Daoud Clarke. 2009. Context-theoretic semantics for natural language: an overview. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 112–119, Athens, Greece. [find](https://scholar.google.com/scholar?q=Daoud+Clarke%2E+2009%2E+Context%2Dtheoretic+semantics+for+natural+language%3A+an+overview%2E+In+Proceedings+of+the+Workshop+on+Geometrical+Models+of+Natural+Language+Semantics%2C+pages+112%E2%80%93119%2C+Athens%2C+Greece%2E)
7. Ruiji Fu, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1199–1209, Baltimore, Maryland. [find](https://scholar.google.com/scholar?q=Ruiji+Fu%2C+Jiang+Guo%2C+Bing+Qin%2C+Wanxiang+Che%2C+Haifeng+Wang%2C+and+Ting+Liu%2E+2014%2E+Learning+semantic+hierarchies+via+word+embeddings%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Compu) [our card](https://md.research.elamur.ai/free/en/papers/fu-2014-learning-semantic-hierarchies-word-embeddings/)
8. Yoav Goldberg and Jon Orwant. 2013. A dataset of syntactic-ngrams over time from a very large corpus of english books. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 241–24… [find](https://scholar.google.com/scholar?q=Yoav+Goldberg+and+Jon+Orwant%2E+2013%2E+A+dataset+of+syntactic%2Dngrams+over+time+from+a+very+large+corpus+of+english+books%2E+In+Second+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2ASEM%29%2C+Volume+)
9. Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics, pages 529–545, Nantes, France. [find](https://scholar.google.com/scholar?q=Marti+A+Hearst%2E+1992%2E+Automatic+acquisition+of+hyponyms+from+large+text+corpora%2E+In+COLING+1992+Volume+2%3A+The+15th+International+Conference+on+Computational+Linguistics%2C+pages+529%E2%80%93545%2C+Nantes%2C+France%2E)
10. David A Jurgens, Peter D Turney, Saif M Mohammad, and Keith J Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume… [find](https://scholar.google.com/scholar?q=David+A+Jurgens%2C+Peter+D+Turney%2C+Saif+M+Mohammad%2C+and+Keith+J+Holyoak%2E+2012%2E+Semeval%2D2012+task+2%3A+Measuring+degrees+of+relational+similarity%2E+In+%2ASEM+2012%3A+The+First+Joint+Conference+on+Lexical+and+Co)
11. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 4(16):359–389. [find](https://scholar.google.com/scholar?q=Lili+Kotlerman%2C+Ido+Dagan%2C+Idan+Szpektor%2C+and+Maayan+Zhitomirsky%2DGeffet%2E+2010%2E+Directional+distributional+similarity+for+lexical+inference%2E+Natural+Language+Engineering%2C+4%2816%29%3A359%E2%80%93389%2E)
12. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–308, Baltimore, Maryland. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Dependency%2Dbased+word+embeddings%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers%29%2C+pages+302%E2%80%93308%2C)
13. Omer Levy, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open ie propositions. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 87–97, Baltimore, Maryland. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Ido+Dagan%2C+and+Jacob+Goldberger%2E+2014%2E+Focused+entailment+graphs+for+open+ie+propositions%2E+In+Proceedings+of+the+Eighteenth+Conference+on+Computational+Natural+Language+Learning%2C+pages+87%E2%80%939)
14. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, pages 768–774, Montréal, Quebec, Canada. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1998%2E+Automatic+retrieval+and+clustering+of+similar+words%2E+In+Proceedings+of+the+36th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+17th+International+Conference+on+C) [our card](https://md.research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/)
15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013a%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations)
16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Gregory+S+Corrado%2C+and+Jeffrey+Dean%2E+2013b%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Proc)
17. Yoshiki Niwa and Yoshihiko Nitta. 1994. Co-occurrence vectors from corpora vs. distance vectors from dictionaries. In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics, Kyoto, Japan. [find](https://scholar.google.com/scholar?q=Yoshiki+Niwa+and+Yoshihiko+Nitta%2E+1994%2E+Co%2Doccurrence+vectors+from+corpora+vs%2E+distance+vectors+from+dictionaries%2E+In+COLING+1994+Volume+1%3A+The+15th+International+Conference+on+Computational+Linguisti)
18. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. [find](https://scholar.google.com/scholar?q=Sebastian+Pad%C3%B3+and+Mirella+Lapata%2E+2007%2E+Dependency%2Dbased+construction+of+semantic+space+models%2E+Computational+Linguistics%2C+33%282%29%3A161%E2%80%93199%2E) [our card](https://md.research.elamur.ai/free/en/papers/pado-2007-dependency-based-construction-semantic-space/)
19. Laura Rimell. 2014. Distributional lexical entailment by topic coherence. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 511–519, Gothenburg, Sweden. [find](https://scholar.google.com/scholar?q=Laura+Rimell%2E+2014%2E+Distributional+lexical+entailment+by+topic+coherence%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+511%E2%80%93519%2C)
20. Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1025– 1036, Dublin, Ireland. [find](https://scholar.google.com/scholar?q=Stephen+Roller%2C+Katrin+Erk%2C+and+Gemma+Boleda%2E+2014%2E+Inclusive+yet+selective%3A+Supervised+distributional+hypernymy+detection%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Conference+on+Computat)
21. Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 38–42, Gothenburg, Sweden. [find](https://scholar.google.com/scholar?q=Enrico+Santus%2C+Alessandro+Lenci%2C+Qin+Lu%2C+and+Sabine+Schulte+im+Walde%2E+2014%2E+Chasing+hypernyms+in+vector+spaces+with+entropy%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Associ)
22. Hinrich Schütze. 1993. Part-of-speech induction from scratch. In Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics, pages 251–258, Columbus, Ohio, USA. [find](https://scholar.google.com/scholar?q=Hinrich+Sch%C3%BCtze%2E+1993%2E+Part%2Dof%2Dspeech+induction+from+scratch%2E+In+Proceedings+of+the+31st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+251%E2%80%93258%2C+Columbus%2C+Ohio%2C+USA%2E)
23. Rion Snow, Daniel Jurafsky, and Andrew Y Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Advances in Neural Information Processing. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Daniel+Jurafsky%2C+and+Andrew+Y+Ng%2E+2005%2E+Learning+syntactic+patterns+for+automatic+hypernym+discovery%2E+In+Advances+in+Neural+Information+Processing%2E)
24. Peter D Turney and Saif M Mohammad. 2014. Experiments with three approaches to recognizing lexical entailment. Natural Language Engineering, pages 1–40. [find](https://scholar.google.com/scholar?q=Peter+D+Turney+and+Saif+M+Mohammad%2E+2014%2E+Experiments+with+three+approaches+to+recognizing+lexical+entailment%2E+Natural+Language+Engineering%2C+pages+1%E2%80%9340%2E)
25. Peter D Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. [find](https://scholar.google.com/scholar?q=Peter+D+Turney%2E+2006%2E+Similarity+of+semantic+relations%2E+Computational+Linguistics%2C+32%283%29%3A379%E2%80%93416%2E) [our card](https://md.research.elamur.ai/free/en/papers/turney-2006-similarity-semantic-relations/)
26. Julie Weeds and David Weir. 2003. A general framework for distributional similarity. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 81–88, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Julie+Weeds+and+David+Weir%2E+2003%2E+A+general+framework+for+distributional+similarity%2E+In+Proceedings+of+the+2003+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+81%E2%80%9388%2C+Sapporo%2C+Ja)
27. Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2249– 2259, Dublin, Ireland. [find](https://scholar.google.com/scholar?q=Julie+Weeds%2C+Daoud+Clarke%2C+Jeremy+Reffin%2C+David+Weir%2C+and+Bill+Keller%2E+2014%2E+Learning+to+distinguish+hypernyms+and+co%2Dhyponyms%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Conference+on+Comp) [our card](https://md.research.elamur.ai/free/en/papers/weeds-2014-learning-distinguish-hypernyms-co-hyponyms/)