---
title: "Efficient Estimation of Word Representations in Vector Space — Mikolov 2013"
description: "The authors from Google propose two simplified log-linear architectures. 32 references, cited by 3,846 papers here."
authors: ["Tomas Mikolov", "Kai Chen", "Greg Corrado", "Jeffrey Dean"]
year: 2013
venue: "arXiv:1301.3781"
source: "https://arxiv.org/abs/1301.3781"
doi: "10.48550/arxiv.1301.3781"
arxiv: "1301.3781"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / Efficient Estimation of Word Representations in Vector Space

# Efficient Estimation of Word Representations in Vector Space

**Authors:** [Tomas Mikolov](https://md.research.elamur.ai/free/en/authors/tomas-mikolov/), [Kai Chen](https://md.research.elamur.ai/free/en/authors/kai-chen/), [Greg Corrado](https://md.research.elamur.ai/free/en/authors/greg-s-corrado/) and [Jeffrey Dean](https://md.research.elamur.ai/free/en/authors/jeff-dean/) · **Year:** 2013 · **Venue:** arXiv:1301.3781 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1301.3781) · [PDF](https://arxiv.org/pdf/1301.3781) · [DOI 10.48550/arxiv.1301.3781](https://doi.org/10.48550/arxiv.1301.3781) · [arXiv 1301.3781](https://arxiv.org/abs/1301.3781)

**Citations:** [3846](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Most NLP systems of that time treated words as separate indices in a vocabulary, so similarity between words was not taken into account at all. Existing neural network models of word vectors were too slow to be trained on billions of words. The authors from Google propose two simplified log-linear architectures. CBOW reconstructs the central word from the averaged vectors of its surrounding words, while Skip-gram does the opposite: given a word, it predicts its neighbors. Neither model has a nonlinear hidden layer. The output is computed via hierarchical softmax over a Huffman tree, and training is distributed in the DistBelief framework with Adagrad. For evaluation, the authors created a new test set of semantic and syntactic analogies. These are solved with simple vector arithmetic, for example Paris − France + Italy ≈ Rome. The main result: high-quality vectors from a corpus of 1.6 billion words can be obtained in less than a day. On analogies they are more accurate than previous neural network approaches, while requiring far less computation. This work laid the foundation of word2vec, and word embeddings became a basic component of almost all subsequent NLP.

## Key points
- **CBOW** takes a context of 4 preceding and 4 following words and ignores word order. The training complexity is Q = N×D + D×log₂(V). **Skip-gram** predicts the words surrounding the current one.
- Hierarchical softmax over a Huffman tree gives short codes to frequent words. For a vocabulary of 1 million words, this is roughly twice as fast as with a balanced binary tree.
- The best vectors in the examples were produced by the Skip-gram model: 300 dimensions, trained on 783 million words. If a relation is specified not by a single example but by the average of ten, test accuracy increases by about 10 percentage points.
- On the Microsoft Sentence Completion Challenge, Skip-gram alone does not outperform LSA. But combined with RNNLM it achieves a new record of 58.9% (the previous one was 55.4%).
- After publication, the authors released multithreaded C++ code that processes billions of words per hour. They also published more than 1.4 million vectors of named entities trained on more than 100 billion words.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cite

Cite as: Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mikolov2013efficient,
  title = {{Efficient Estimation of Word Representations in Vector Space}},
  author = {Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean},
  year = {2013},
  doi = {10.48550/arxiv.1301.3781},
  eprint = {1301.3781},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1301.3781}
}
```

- [← previous Distributed Representations of Words and Phrases and their Compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
- [next → Linguistic Regularities in Continuous Space Word Representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Y. Bengio, R. Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003. [our card](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/)
2. Y. Bengio, Y. LeCun. Scaling learning algorithms towards AI. In: Large-Scale Kernel Machines, MIT Press, 2007. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2C+Y%2E+LeCun%2E+Scaling+learning+algorithms+towards+AI%2E+In%3A+Large%2DScale+Kernel+Machines%2C+MIT+Press%2C+2007%2E)
3. T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. Large language models in machine translation. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007. [our card](https://md.research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/)
4. R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008. [find](https://scholar.google.com/scholar?q=R%2E+Collobert+and+J%2E+Weston%2E+A+Unified+Architecture+for+Natural+Language+Processing%3A+Deep+Neural+Networks+with+Multitask+Learning%2E+In+International+Conference+on+Machine+Learning%2C+ICML%2C+2008%2E)
5. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493-2537, 2011. [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
6. J. Dean, G.S. Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, A. Y. Ng., Large Scale Distributed Deep Networks, NIPS, 2012. [find](https://scholar.google.com/scholar?q=J%2E+Dean%2C+G%2ES%2E+Corrado%2C+R%2E+Monga%2C+K%2E+Chen%2C+M%2E+Devin%2C+Q%2EV%2E+Le%2C+M%2EZ%2E+Mao%2C+M%2EA%2E+Ranzato%2C+A%2E+Senior%2C+P%2E+Tucker%2C+K%2E+Yang%2C+A%2E+Y%2E+Ng%2E%2C+Large+Scale+Distributed+Deep+Networks%2C+NIPS%2C+2012%2E)
7. J.C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011. [find](https://scholar.google.com/scholar?q=J%2EC%2E+Duchi%2C+E%2E+Hazan%2C+and+Y%2E+Singer%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+Machine+Learning+Research%2C+2011%2E)
8. J. Elman. Finding Structure in Time. Cognitive Science, 14, 179-211, 1990. [find](https://scholar.google.com/scholar?q=J%2E+Elman%2E+Finding+Structure+in+Time%2E+Cognitive+Science%2C+14%2C+179%2D211%2C+1990%2E)
9. Eric H. Huang, R. Socher, C. D. Manning and Andrew Y. Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In: Proc. Association for Computational Linguistics, 2012. [our card](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)
10. G.E. Hinton, J.L. McClelland, D.E. Rumelhart. Distributed representations. In: Parallel distributed processing: Explorations in the microstructure of cognition. Volume 1: Foundations, MIT Press, 1986. [find](https://scholar.google.com/scholar?q=G%2EE%2E+Hinton%2C+J%2EL%2E+McClelland%2C+D%2EE%2E+Rumelhart%2E+Distributed+representations%2E+In%3A+Parallel+distributed+processing%3A+Explorations+in+the+microstructure+of+cognition%2E+Volume+1%3A+Foundations%2C+MIT+Press%2C+1986%2E)
11. D.A. Jurgens, S.M. Mohammad, P.D. Turney, K.J. Holyoak. Semeval-2012 task 2: Measuring degrees of relational similarity. In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012. [find](https://scholar.google.com/scholar?q=D%2EA%2E+Jurgens%2C+S%2EM%2E+Mohammad%2C+P%2ED%2E+Turney%2C+K%2EJ%2E+Holyoak%2E+Semeval%2D2012+task+2%3A+Measuring+degrees+of+relational+similarity%2E+In%3A+Proceedings+of+the+6th+International+Workshop+on+Semantic+Evaluation+%28SemEv)
12. A.L. Maas, R.E. Daly, P.T. Pham, D. Huang, A.Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of ACL, 2011. [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
13. T. Mikolov. Language Modeling for Speech Recognition in Czech, Masters thesis, Brno University of Technology, 2007. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2E+Language+Modeling+for+Speech+Recognition+in+Czech%2C+Masters+thesis%2C+Brno+University+of+Technology%2C+2007%2E)
14. T. Mikolov, J. Kopecký, L. Burget, O. Glembek and J. Černocký. Neural network based language models for higly inflective languages, In: Proc. ICASSP 2009. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+J%2E+Kopecky%CC%81%2C+L%2E+Burget%2C+O%2E+Glembek+and+J%2E+C%CC%8Cernocky%CC%81%2E+Neural+network+based+language+models+for+higly+inflective+languages%2C+In%3A+Proc%2E+ICASSP+2009%2E)
15. T. Mikolov, M. Karafiát, L. Burget, J. Černocký, S. Khudanpur. Recurrent neural network based language model, In: Proceedings of Interspeech, 2010. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+M%2E+Karafia%CC%81t%2C+L%2E+Burget%2C+J%2E+C%CC%8Cernocky%CC%81%2C+S%2E+Khudanpur%2E+Recurrent+neural+network+based+language+model%2C+In%3A+Proceedings+of+Interspeech%2C+2010%2E)
16. T. Mikolov, S. Kombrink, L. Burget, J. Černocký, S. Khudanpur. Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+S%2E+Kombrink%2C+L%2E+Burget%2C+J%2E+C%CC%8Cernocky%CC%81%2C+S%2E+Khudanpur%2E+Extensions+of+recurrent+neural+network+language+model%2C+In%3A+Proceedings+of+ICASSP+2011%2E)
17. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, J. Černocký. Empirical Evaluation and Combination of Advanced Language Modeling Techniques, In: Proceedings of Interspeech, 2011. The code is available at [https://code.google.com/p/word2vec/](https://code.google.com/p/word2vec/) [↗](https://code.google.com/p/word2vec/)
18. T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Černocký. Strategies for Training Large Scale Neural Network Language Models, In: Proc. Automatic Speech Recognition and Understanding, 2011. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+A%2E+Deoras%2C+D%2E+Povey%2C+L%2E+Burget%2C+J%2E+C%CC%8Cernocky%CC%81%2E+Strategies+for+Training+Large+Scale+Neural+Network+Language+Models%2C+In%3A+Proc%2E+Automatic+Speech+Recognition+and+Understanding%2C+2011%2E)
19. T. Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2E+Statistical+Language+Models+based+on+Neural+Networks%2E+PhD+thesis%2C+Brno+University+of+Technology%2C+2012%2E)
20. T. Mikolov, W.T. Yih, G. Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL HLT 2013. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/)
21. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed Representations of Words and Phrases and their Compositionality. Accepted to NIPS 2013. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
22. A. Mnih, G. Hinton. Three new graphical models for statistical language modelling. ICML, 2007. [find](https://scholar.google.com/scholar?q=A%2E+Mnih%2C+G%2E+Hinton%2E+Three+new+graphical+models+for+statistical+language+modelling%2E+ICML%2C+2007%2E)
23. A. Mnih, G. Hinton. A Scalable Hierarchical Distributed Language Model. Advances in Neural Information Processing Systems 21, MIT Press, 2009. [find](https://scholar.google.com/scholar?q=A%2E+Mnih%2C+G%2E+Hinton%2E+A+Scalable+Hierarchical+Distributed+Language+Model%2E+Advances+in+Neural+Information+Processing+Systems+21%2C+MIT+Press%2C+2009%2E)
24. A. Mnih, Y.W. Teh. A fast and simple algorithm for training neural probabilistic language models. ICML, 2012. [find](https://scholar.google.com/scholar?q=A%2E+Mnih%2C+Y%2EW%2E+Teh%2E+A+fast+and+simple+algorithm+for+training+neural+probabilistic+language+models%2E+ICML%2C+2012%2E)
25. F. Morin, Y. Bengio. Hierarchical Probabilistic Neural Network Language Model. AISTATS, 2005. [find](https://scholar.google.com/scholar?q=F%2E+Morin%2C+Y%2E+Bengio%2E+Hierarchical+Probabilistic+Neural+Network+Language+Model%2E+AISTATS%2C+2005%2E)
26. D. E. Rumelhart, G. E. Hinton, R. J. Williams. Learning internal representations by backpropagating errors. Nature, 323:533.536, 1986. [find](https://scholar.google.com/scholar?q=D%2E+E%2E+Rumelhart%2C+G%2E+E%2E+Hinton%2C+R%2E+J%2E+Williams%2E+Learning+internal+representations+by+backpropagating+errors%2E+Nature%2C+323%3A533%2E536%2C+1986%2E)
27. H. Schwenk. Continuous space language models. Computer Speech and Language, vol. 21, 2007. [find](https://scholar.google.com/scholar?q=H%2E+Schwenk%2E+Continuous+space+language+models%2E+Computer+Speech+and+Language%2C+vol%2E+21%2C+2007%2E)
28. R. Socher, E.H. Huang, J. Pennington, A.Y. Ng, and C.D. Manning. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS, 2011. [find](https://scholar.google.com/scholar?q=R%2E+Socher%2C+E%2EH%2E+Huang%2C+J%2E+Pennington%2C+A%2EY%2E+Ng%2C+and+C%2ED%2E+Manning%2E+Dynamic+Pooling+and+Unfolding+Recursive+Autoencoders+for+Paraphrase+Detection%2E+In+NIPS%2C+2011%2E)
29. J. Turian, L. Ratinov, Y. Bengio. Word Representations: A Simple and General Method for Semi-Supervised Learning. In: Proc. Association for Computational Linguistics, 2010. [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
30. P. D. Turney. Measuring Semantic Similarity by Latent Relational Analysis. In: Proc. International Joint Conference on Artificial Intelligence, 2005. [find](https://scholar.google.com/scholar?q=P%2E+D%2E+Turney%2E+Measuring+Semantic+Similarity+by+Latent+Relational+Analysis%2E+In%3A+Proc%2E+International+Joint+Conference+on+Artificial+Intelligence%2C+2005%2E)
31. A. Zhila, W.T. Yih, C. Meek, G. Zweig, T. Mikolov. Combining Heterogeneous Models for Measuring Relational Similarity. NAACL HLT 2013. [find](https://scholar.google.com/scholar?q=A%2E+Zhila%2C+W%2ET%2E+Yih%2C+C%2E+Meek%2C+G%2E+Zweig%2C+T%2E+Mikolov%2E+Combining+Heterogeneous+Models+for+Measuring+Relational+Similarity%2E+NAACL+HLT+2013%2E)
32. G. Zweig, C.J.C. Burges. The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011. [find](https://scholar.google.com/scholar?q=G%2E+Zweig%2C+C%2EJ%2EC%2E+Burges%2E+The+Microsoft+Research+Sentence+Completion+Challenge%2C+Microsoft+Research+Technical+Report+MSR%2DTR%2D2011%2D129%2C+2011%2E)