---
title: "Efficient Non-parametric Estimation of — Neelakantan 2014"
description: "Standard word embeddings such as Skip-gram assign one vector to each word type."
authors: ["Arvind Neelakantan", "Jeevan Shankar", "Alexandre Passos", "Andrew McCallum"]
year: 2014
venue: "Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/D14-1113/"
doi: "10.3115/v1/d14-1113"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/neelakantan-2014-efficient-non-parametric-estimation-multiple/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/neelakantan-2014-efficient-non-parametric-estimation-multiple/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

# Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

**Authors:** [Arvind Neelakantan](https://md.research.elamur.ai/free/en/authors/arvind-neelakantan/), Jeevan Shankar, [Alexandre Passos](https://md.research.elamur.ai/free/en/authors/alexandre-passos/), [Andrew McCallum](https://md.research.elamur.ai/free/en/authors/andrew-mccallum/) · **Year:** 2014 · **Venue:** Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D14-1113/) · [PDF](https://aclanthology.org/D14-1113.pdf) · [DOI 10.3115/v1/d14-1113](https://doi.org/10.3115/v1/d14-1113) · [OpenAlex](https://openalex.org/W2162456950)

**Citations:** 442 citations (openalex-snapshot · 2026-10-14) · [151](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard word embeddings such as Skip-gram assign one vector to each word type. A polysemous word like *plant* then ends up as a blend of its meanings, and through the triangle inequality, unrelated words tied to different senses (e.g., *pollen* and *refinery*) get pulled toward each other. The authors extend Skip-gram with negative sampling into Multiple-sense Skip-gram (MSSG). Each word keeps a global vector plus several sense vectors, each with its own context cluster center. During online training, a token's context is summarized as the average of its neighbours' global vectors. The nearest cluster picks the sense, and only that sense vector is updated. Sense discrimination and embedding learning therefore happen jointly, not as a separate clustering step beforehand. A non-parametric variant, NP-MSSG, opens new senses on the fly, so the number of senses varies per word. The models set a new state of the art on the in-context similarity benchmark SCWS. Training on nearly 1 billion tokens takes under 6 hours on one machine, about 27x faster than the earlier multi-sense approach of Huang et al. (2012).

## Key points
- On SCWS (Spearman ρ×100), MSSG-300d scores 69.3 with avgSimC, versus 65.7 for Huang et al. (2012). NP-MSSG-300d reaches 69.1 and is best on globalSim, avgSim and localSim.
- On WordSim353, MSSG 300d with avgSim gets 70.9, slightly above Skip-gram-300d (70.4) and well above Huang et al. (64.2).
- On the Google Analogy task, MSSG and NP-MSSG reach 64% accuracy, compared with 12% for Huang et al. and 67% for single-sense Skip-gram.
- Multiple senses were learned for the same ~6,000 words used by Huang et al. When this is extended to the top 30,000 frequent words (300d, 3 senses), SCWS avgSimC is still 69.2.
- Global vectors, sense vectors, cluster centers and training code are publicly released. By comparison, Huang et al.'s method needed about a week of computation for a similar setup.

## Interesting ideas
- **Joint clustering and learning:** the sense of each token is chosen with the model's current parameters, so the clustering improves as the vectors improve. This avoids a costly two-stage pipeline and could apply to any task where latent categories and representations are learned together.
- **Distance-triggered cluster creation:** following facility location (Meyerson, 2001), a new sense is created with probability proportional to how far a context is from the nearest existing sense. This simple online rule removes the need to fix the number of clusters and could serve other streaming clustering problems.
- **Separate global vectors for contexts:** contexts are represented by single-sense global vectors while target words get sense-specific ones. This keeps context representations stable and cheap, a design that suits other multi-prototype models.
- **Evaluating with and without context:** the paper reports several similarity measures (globalSim, avgSim, avgSimC, localSim) side by side. This shows where sense modelling helps (in-context similarity) and where it hurts (context-free analogies). The idea is useful when building benchmarks for ambiguous words in any language, for example a Ukrainian SCWS-style dataset.
- **Triangle-inequality argument:** this geometric point explains concretely why one vector per word distorts the space. It is a good diagnostic lens when checking odd nearest neighbours in existing embeddings.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

151 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 146 not yet.

- [Deep Contextualized Pairwise Semantic Similarity for Arabic Language Questions](https://md.research.elamur.ai/free/en/papers/al-2019-bataineh-deep-contextualized-pairwise-semantic-similarity/)
- [A Unified Model for Word Sense Representation and Disambiguation](https://md.research.elamur.ai/free/en/papers/chen-2014-unified-model-word-sense-representation/)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://md.research.elamur.ai/free/en/papers/faruqui-2016-problems-evaluation-word-embeddings-word/)
- [Ontology-Aware Token Embeddings for Prepositional Phrase Attachment](https://md.research.elamur.ai/free/en/papers/onto-2017-dasigi-ontology-aware-token-embeddings-prepositional/)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{neelakantan2014efficient,
  title = {{Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space}},
  author = {Neelakantan, Arvind and Shankar, Jeevan and Passos, Alexandre and McCallum, Andrew},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1113},
  url = {https://doi.org/10.3115/v1/d14-1113}
}
```

- [← previous Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors](https://md.research.elamur.ai/free/en/papers/baroni-2014-don-t-count-predict-systematic/)
- [next → Etymological Wordnet: Tracing The History of Words](https://md.research.elamur.ai/free/en/papers/lex-2014-demelo-etymological-wordnet/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

## References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring Continuous Word Representations for Dependency Parsing. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Mohit+Bansal%2C+Kevin+Gimpel%2C+and+Karen+Livescu%2E+2014%2E+Tailoring+Continuous+Word+Representations+for+Dependency+Parsing%2E+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bansal-2014-tailoring-continuous-word-representations-dependency/)
2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research (JMLR). [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Re%CC%81jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Jauvin%2E+2003%2E+A+neural+probabilistic+language+model%2E+Journal+of+Machine+Learning+Research+%28JMLR%29%2E)
3. Peter F. Brown, Peter V. Desouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based N-gram models of natural language Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peter+F%2E+Brown%2C+Peter+V%2E+Desouza%2C+Robert+L%2E+Mercer%2C+Vincent+J%2E+Della+Pietra%2C+and+Jenifer+C%2E+Lai%2E+1992%2E+Class%2Dbased+N%2Dgram+models+of+natural+language+Computational+Linguistics%2E)
4. Ronan Collobert and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. International Conference on Machine learning (ICML). [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+Unified+Architecture+for+Natural+Language+Processing%3A+Deep+Neural+Networks+with+Multitask+Learning%2E+International+Conference+on+Machine+learning+%28ICML%29%2E)
5. Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. 2011. Multi-View Learning of Word Embeddings via CCA. Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Paramveer+S%2E+Dhillon%2C+Dean+Foster%2C+and+Lyle+Ungar%2E+2011%2E+Multi%2DView+Learning+of+Word+Embeddings+via+CCA%2E+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E)
6. John Duchi, Elad Hazan, and Yoram Singer 2011. Adaptive sub- gradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (JMLR). [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer+2011%2E+Adaptive+sub%2D+gradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+Machine+Learning+Research+%28JMLR%29%2E)
7. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. International Conference on World Wide Web (WWW). [find](https://scholar.google.com/scholar?q=Lev+Finkelstein%2C+Evgeniy+Gabrilovich%2C+Yossi+Matias%2C+Ehud+Rivlin%2C+Zach+Solan%2C+Gadi+Wolfman%2C+and+Eytan+Ruppin%2E+2001%2E+Placing+search+in+context%3A+the+concept+revisited%2E+International+Conference+on+World+W)
8. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. International Joint Conference on Artificial Intelligence (IJCAI). [find](https://scholar.google.com/scholar?q=Evgeniy+Gabrilovich+and+Shaul+Markovitch%2E+2007%2E+Computing+semantic+relatedness+using+wikipedia%2Dbased+explicit+semantic+analysis%2E+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%29%2E)
9. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. Association of Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Eric+H%2E+Huang%2C+Richard+Socher%2C+Christopher+D%2E+Manning%2C+and+Andrew+Y%2E+Ng%2E+2012%2E+Improving+Word+Representations+via+Global+Context+and+Multiple+Word+Prototypes%2E+Association+of+Computational+Linguistics+) [our card](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)
10. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple Semi-supervised Dependency Parsing. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Terry+Koo%2C+Xavier+Carreras%2C+and+Michael+Collins%2E+2008%2E+Simple+Semi%2Dsupervised+Dependency+Parsing%2E+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/koo-2008-simple-semi-supervised-dependency-parsing/)
11. Quoc V. Le and Tomas Mikolov. 2014 Distributed Representations of Sentences and Documents. International Conference on Machine Learning (ICML) [find](https://scholar.google.com/scholar?q=Quoc+V%2E+Le+and+Tomas+Mikolov%2E+2014+Distributed+Representations+of+Sentences+and+Documents%2E+International+Conference+on+Machine+Learning+%28ICML%29) [our card](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/)
12. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011 Learning Word Vectors for Sentiment Analysis Association for Computational Linguistics (ACL) [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011+Learning+Word+Vectors+for+Sentiment+Analysis+Association+for+Computational+Linguistics+%28ACL%29) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
13. Adam Meyerson. 2001 IEEE Symposium on Foundations of Computer Science. International Conference on Machine Learning (ICML) [find](https://scholar.google.com/scholar?q=Adam+Meyerson%2E+2001+IEEE+Symposium+on+Foundations+of+Computer+Science%2E+International+Conference+on+Machine+Learning+%28ICML%29)
14. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. Workshop at International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013a%2E+Efficient+Estimation+of+Word+Representations+in+Vector+Space%2E+Workshop+at+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013b%2E+Distributed+Representations+of+Words+and+Phrases+and+their+Compositionality%2E+Advances+in+Neural+Information+Processing+S)
16. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013c. Exploiting Similarities among Languages for Machine Translation. arXiv. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Quoc+V%2E+Le%2C+and+Ilya+Sutskever%2E+2013c%2E+Exploiting+Similarities+among+Languages+for+Machine+Translation%2E+arXiv%2E)
17. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). [find](https://scholar.google.com/scholar?q=Scott+Miller%2C+Jethran+Guinness%2C+and+Alex+Zamanian%2E+2004%2E+Name+tagging+with+word+clusters+and+discriminative+training%2E+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Lan) [our card](https://md.research.elamur.ai/free/en/papers/miller-2004-name-tagging-word-clusters-discriminative/)
18. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. International Conference on Machine learning (ICML). [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Geoffrey+Hinton%2E+2007%2E+Three+new+graphical+models+for+statistical+language+modelling%2E+International+Conference+on+Machine+learning+%28ICML%29%2E)
19. Arvind Neelakantan and Michael Collins. 2014. Learning Dictionaries for Named Entity Recognition using Minimal Supervision. European Chapter of the Association for Computational Linguistics (EACL). [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan+and+Michael+Collins%2E+2014%2E+Learning+Dictionaries+for+Named+Entity+Recognition+using+Minimal+Supervision%2E+European+Chapter+of+the+Association+for+Computational+Linguistics+%28EACL%29%2E)
20. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon Infused Phrase Embeddings for Named Entity Resolution. Conference on Natural Language Learning (CoNLL). [find](https://scholar.google.com/scholar?q=Alexandre+Passos%2C+Vineet+Kumar%2C+and+Andrew+McCallum%2E+2014%2E+Lexicon+Infused+Phrase+Embeddings+for+Named+Entity+Resolution%2E+Conference+on+Natural+Language+Learning+%28CoNLL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/passos-2014-lexicon-infused-phrase-embeddings-named/)
21. Lev Ratinov and Dan Roth. 2009. Design Challenges and Misconceptions in Named Entity Recognition. Conference on Natural Language Learning (CoNLL). [find](https://scholar.google.com/scholar?q=Lev+Ratinov+and+Dan+Roth%2E+2009%2E+Design+Challenges+and+Misconceptions+in+Named+Entity+Recognition%2E+Conference+on+Natural+Language+Learning+%28CoNLL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/ratinov-2009-design-challenges-misconceptions-named-entity/)
22. Siva Reddy, Ioannis P. Klapaftis, and Diana McCarthy. 2011. Dynamic and Static Prototype Vectors for Semantic Composition. International Joint Conference on Artificial Intelligence (IJCNLP). [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Ioannis+P%2E+Klapaftis%2C+and+Diana+McCarthy%2E+2011%2E+Dynamic+and+Static+Prototype+Vectors+for+Semantic+Composition%2E+International+Joint+Conference+on+Artificial+Intelligence+%28IJCNLP%29%2E)
23. Joseph Reisinger and Raymond J. Mooney. 2010a. Multi-prototype vector-space models of word meaning. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) [find](https://scholar.google.com/scholar?q=Joseph+Reisinger+and+Raymond+J%2E+Mooney%2E+2010a%2E+Multi%2Dprototype+vector%2Dspace+models+of+word+meaning%2E+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies) [our card](https://md.research.elamur.ai/free/en/papers/reisinger-2010-multi-prototype-vector-space-models/)
24. Joseph Reisinger and Raymond Mooney. 2010b. A mixture model with sharing for lexical semantics. Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Joseph+Reisinger+and+Raymond+Mooney%2E+2010b%2E+A+mixture+model+with+sharing+for+lexical+semantics%2E+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
25. Cyrus Shaoul and Chris Westbury. 2010. The Westbury lab wikipedia corpus. [find](https://scholar.google.com/scholar?q=Cyrus+Shaoul+and+Chris+Westbury%2E+2010%2E+The+Westbury+lab+wikipedia+corpus%2E)
26. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011 Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Eric+H%2E+Huang%2C+Jeffrey+Pennington%2C+Andrew+Y%2E+Ng%2C+and+Christopher+D%2E+Manning%2E+2011+Dynamic+Pooling+and+Unfolding+Recursive+Autoencoders+for+Paraphrase+Detection%2E+Advances+in+Neural+Info)
27. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Oscar+Ta%CC%88ckstro%CC%88m%2C+Ryan+McDonald%2C+and+Jakob+Uszkoreit%2E+2012%2E+Cross%2Dlingual+Word+Clusters+for+Direct+Transfer+of+Linguistic+Structure%2E+North+American+Chapter+of+the+Association+for+Computational+Lingui) [our card](https://md.research.elamur.ai/free/en/papers/tackstrom-2012-cross-lingual-word-clusters-direct/)
28. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word Representations: A Simple and General Method for Semi-Supervised Learning. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Joseph+Turian%2C+Lev+Ratinov%2C+and+Yoshua+Bengio%2E+2010%2E+Word+Representations%3A+A+Simple+and+General+Method+for+Semi%2DSupervised+Learning%2E+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
29. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual Word Embeddings for Phrase-Based Machine Translation. Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Will+Y%2E+Zou%2C+Richard+Socher%2C+Daniel+Cer%2C+and+Christopher+D%2E+Manning%2E+2013%2E+Bilingual+Word+Embeddings+for+Phrase%2DBased+Machine+Translation%2E+Empirical+Methods+in+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/zou-2013-bilingual-word-embeddings-phrase-based/)