---
title: "Word embeddings — definition and key papers"
description: "Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space."
canonical: "https://research.elamur.ai/free/en/concepts/word-embeddings/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/word-embeddings/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Word embeddings

# 🧮 Word embeddings

Concept · 30 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Word embeddings are dense, real-valued vectors that represent the words of a vocabulary as points in a shared geometric space. They are usually learned from co-occurrence patterns in large unlabeled corpora, so that words used in similar contexts end up close to one another.

Also: word vectors, distributed word representations, continuous-space word representations, vector space word representations, static embeddings

## What it is

Classical NLP systems treat each word as a one-hot symbol, an index with no relation to any other index. Under this encoding, rare words receive poorly estimated parameters and unseen words receive none ([Turian et al. 2010](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)). A word embedding replaces the symbol with a vector of a few dozen to a few hundred dimensions. Similarity between words then becomes a geometric quantity, typically cosine similarity.

Typical sizes vary widely. Huang et al.'s global-context model uses 50 dimensions over a 30,000-word vocabulary ([Huang et al. 2012](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)). The public word2vec model trained on Google News holds roughly 3 million English words and phrases, each a 300-dimensional vector ([Bolukbasi et al. 2016](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)).

The idea rests on the distributional hypothesis, associated with Harris (1954) and Firth (1957): a word's meaning is reflected in the company it keeps. Embeddings turn that intuition into a trainable object. They can be learned once from raw text and then plugged into almost any supervised system as extra input features ([Turian et al. 2010](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)).

## The key idea

A widely used training mechanism is the skip-gram model with negative sampling. In this model, each word must predict the words that appear near it ([Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/); [Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)). The learned space is commonly evaluated with vector arithmetic on analogies ([Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/)).

```
J(w, c) = log σ(v′_c · v_w) + Σ_{i=1..k} log σ(−v′_{n_i} · v_w)

answer(a : b :: c : ?) = argmax_{x ∉ {a, b, c}} cos(v_x, v_b − v_a + v_c)
```

In the first line, w is a target word and c is a word observed inside its context window. v_w is the "input" vector of w, and v′_c is the "output" vector of the context word c. The function σ is the logistic sigmoid, and the dot product measures how compatible the two vectors are. The sum runs over k "negative" words n_i, drawn at random from a noise distribution. Training raises J for each observed (w, c) pair. This pulls true context pairs together and pushes random pairs apart.

In the second line, a, b and c are the given words of an analogy, and v denotes their vectors. The argmax runs over vocabulary words x other than a, b and c. These three are excluded because one of them would otherwise usually win. cos is cosine similarity. The answer is the word whose vector lies nearest to the offset vector v_b − v_a + v_c.

The striking empirical finding was that linguistic relations behave roughly like constant directions in the space ([Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/)).

Count-based methods reach similar spaces from a different starting point. GloVe fits word vectors directly to global co-occurrence statistics ([Pennington et al. 2014](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)). Sparse PPMI vectors built from the same corpus and window as skip-gram recover a comparable amount of analogical structure. The parity is clearest when analogies are scored with the multiplicative rule described next ([Levy & Goldberg 2014](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)). The same analysis showed that the additive analogy rule is algebraically a combination of three ordinary word similarities. On that basis it proposed a multiplicative variant, 3COSMUL ([Levy & Goldberg 2014](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)). Levy and Goldberg concluded that neural embeddings mainly preserve regularities already present in co-occurrence counts. Under the additive rule, however, the explicit vectors trail somewhat ([Levy & Goldberg 2014](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)).

## History and founding papers

Before 2013, several families of representation coexisted. On the non-neural side, the main options were count-based vectors such as LSA, obtained by factorizing a co-occurrence matrix ([Faruqui & Dyer 2014](https://md.research.elamur.ai/free/en/papers/faruqui-2014-improving-vector-space-word-representations/)), and Brown clusters ([Turian et al. 2010](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)). Neural language models also learned vectors as a by-product. Notable examples are Collobert and Weston's (2008) ranking-loss model and Mnih and Hinton's (2009) HLBL, a hierarchical log-bilinear model ([Turian et al. 2010](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)). Turian and colleagues compared these representations under controlled conditions on named-entity recognition and chunking. Every representation improved accuracy. Brown clusters handled rare words better than the embeddings, which mattered most for NER ([Turian et al. 2010](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)).

Other models of the period:

- Maas et al. added a supervised sentiment term so that vectors also captured polarity ([Maas et al. 2011](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)).
- Huang et al. combined local window context with document context and learned multiple vectors per word for ambiguous words. They trained on Wikipedia text of about 990M tokens ([Huang et al. 2012](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)).

The turning point came in 2013. Vectors taken from a recurrent language model answered 39.6% of a new 8000-question syntactic analogy set. The strongest LSA model, built from identical training data, reached only 16.5% ([Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/)). The word2vec papers then introduced the simpler CBOW and skip-gram architectures, which could be trained cheaply on very large corpora ([Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)). They also added negative sampling, frequent-word subsampling and phrase vectors ([Mikolov et al. 2013](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)). GloVe followed with a count-based global objective ([Pennington et al. 2014](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)).

A parallel line of work addressed morphology. Luong et al. built word vectors from morpheme vectors with a recursive network, with the largest gains on rare words ([Luong et al. 2013](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/)). fastText later represented each word as a bag of character n-grams ([Bojanowski et al. 2017](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)). That approach was scaled to 157 languages using Wikipedia plus Common Crawl ([Grave et al. 2018](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/)). It was also improved for English with phrases, position-dependent weights and deduplicated training on 630B words of Common Crawl ([Mikolov et al. 2017](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/)).

| Approach | Year | Signal used |
| --- | --- | --- |
| C&W / HLBL (compared by Turian et al., 2010) | 2008–2009 | language-model by-product |
| word2vec (CBOW, skip-gram) | 2013 | local context prediction |
| GloVe | 2014 | global co-occurrence counts |
| fastText | 2017 | context prediction plus subwords |
| ELMo | 2018 | deep bidirectional language model |

In 2018, contextualized representations such as ELMo became prominent. They assign each token a vector that depends on its sentence ([Peters et al. 2018](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)). Language models built with LSTM, Transformer and gated-CNN backbones all beat GloVe on tasks such as NLI, semantic role labeling, parsing and NER ([Peters et al. 2018](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)).

## How it is used today

The most common use is as an input layer for task models. Pretrained vectors used as the only input let recurrent taggers outperform feature-rich CRFs on aspect extraction ([Liu et al. 2015](https://md.research.elamur.ai/free/en/papers/liu-2015-fine-grained-opinion-mining-recurrent/)). Embeddings can also be specialized for a task. Sentiment-specific vectors were trained on about 10 million emoticon-labelled tweets. On SemEval 2013 positive/negative classification they reached 84.89% macro-F1, roughly matching the 84.70% of the top hand-engineered system, Mohammad et al. (2013) ([Tang et al. 2014](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/)).

Cross-lingual mapping is a second major application. The usual tool is a linear map between monolingual spaces:

- For English-to-Spanish word translation with skip-gram vectors trained on WMT11 text, two changes raised P@1 from about 35% to about 41%. The changes were normalizing the vectors and restricting the map to be orthogonal; the 35% baseline is Mikolov's unconstrained linear map ([Xing et al. 2015](https://md.research.elamur.ai/free/en/papers/xing-2015-normalized-word-embedding-orthogonal-transform/)).
- A later framework adds orthogonality, length normalization and mean centering to the least-squares mapping objective, and recasts earlier methods as special cases. It yields an exact closed-form solution, W = VUᵀ, where UΣVᵀ is the SVD of ZᵀX and X and Z hold the source and target vectors of the dictionary pairs ([Artetxe et al. 2016](https://md.research.elamur.ai/free/en/papers/artetxe-2016-learning-principled-bilingual-mappings-word/)).
- CCA on translation pairs can even improve monolingual vectors ([Faruqui & Dyer 2014](https://md.research.elamur.ai/free/en/papers/faruqui-2014-improving-vector-space-word-representations/)).
- Bilingual embeddings have served as a phrase-pair similarity feature in phrase-based translation ([Zou et al. 2013](https://md.research.elamur.ai/free/en/papers/zou-2013-bilingual-word-embeddings-phrase-based/)).

Supervision has since shrunk toward zero. Self-learning works from a seed of 25 word pairs or a list of numerals ([Artetxe et al. 2017](https://md.research.elamur.ai/free/en/papers/artetxe-2017-learning-bilingual-word-embeddings-almost/)). A fully unsupervised method was tested on an extended version of the Dinu et al. dataset. It reached 33.50% English–Finnish translation accuracy in its best of ten runs and 32.63% on average, where earlier adversarial unsupervised methods stayed below 2% ([Artetxe et al. 2018](https://md.research.elamur.ai/free/en/papers/artetxe-2018-vecmap-unsupervised/)).

Embeddings also serve as measurement instruments. Decade-by-decade vectors aligned with orthogonal Procrustes revealed two regularities in semantic change. Frequent words change more slowly, and polysemous words change faster once frequency is controlled for ([Hamilton et al. 2016](https://md.research.elamur.ai/free/en/papers/hamilton-2016-diachronic-word-embeddings-reveal-statistical/)). The MoverScore metric compares generated texts with references using Earth Mover's Distance over contextualized embeddings, chiefly BERT ([Zhao et al. 2019](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)). On the engineering side, product quantization shrinks fastText classifiers to roughly a hundredth of their memory with little loss in accuracy ([Joulin et al. 2016](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)).

## Open questions

Evaluation remains unsettled. When six embedding methods were trained under matched conditions, their ranking changed with the criterion used. Downstream scores also proved an unreliable proxy for general quality ([Schnabel et al. 2015](https://md.research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/)). The same study found that vectors encode word frequency strongly enough to distort cosine similarity ([Schnabel et al. 2015](https://md.research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/)). Analogy accuracy, the classic benchmark, partly reflects choices in the scoring rule ([Levy & Goldberg 2014](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/)).

Social bias is a second concern. Gender is largely captured by a single direction in word2vec space. Hard debiasing lowered the share of generated analogies that crowd workers judged stereotypical from about 19% to about 6% ([Bolukbasi et al. 2016](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)). For contextual models, standard cosine-based tests miss biases that probability-based probes reveal ([Kurita et al. 2019](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/)).

Finally, the relation between static and contextual vectors is still being mapped. Static vectors conflate word senses, which motivated multi-prototype models ([Huang et al. 2012](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)). Probes show that contextual spaces encode dependency trees geometrically, as squared distances after a learned linear transformation ([Hewitt & Manning 2019](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/)). Yet their advantage over word-level baselines is much smaller for semantic than for syntactic phenomena ([Tenney et al. 2019](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)). What exactly context adds beyond word identity remains an active research question.

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-23.

- [GloVe: Global Vectors for Word Representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/) — Jeffrey Pennington, Richard Socher, Christopher Manning · 2014 · 5,962 citations
- [Learning Word Vectors for Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/) — Andrew L. Maas, Raymond E. Daly, Peter T. Pham et al. · 2011 · 4,272 citations
- [Distributed Representations of Words and Phrases and their Compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/) — Tomas Mikolov, Ilya Sutskever, Kai Chen et al. · 2013 · 4,233 citations
- [Efficient Estimation of Word Representations in Vector Space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/) — Tomas Mikolov, Kai Chen, Greg Corrado et al. · 2013 · 3,846 citations
- [Linguistic Regularities in Continuous Space Word Representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/) — Tomas Mikolov, Wen-tau Yih, Geoffrey Zweig · 2013 · 3,482 citations
- [Deep Contextualized Word Representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/) — Matthew E. Peters, Mark Neumann, Mohit Iyyer et al. · 2018 · 3,240 citations
- [Word Representations: A Simple and General Method for Semi-Supervised Learning](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/) — Joseph Turian, Lev-Arie Ratinov, Yoshua Bengio · 2010 · 2,421 citations
- [Enriching Word Vectors with Subword Information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/) — Piotr Bojanowski, Edouard Grave, Armand Joulin et al. · 2017 · 2,218 citations
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/) — Tolga Bolukbasi, Kai-Wei Chang, James Zou et al. · 2016 · 2,038 citations
- [Improving Word Representations via Global Context and Multiple Word Prototypes](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/) — Eric Huang, Richard Socher, Christopher Manning et al. · 2012 · 1,349 citations
- [Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/) — Duyu Tang, Furu Wei, Nan Yang et al. · 2014 · 1,341 citations
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://md.research.elamur.ai/free/en/papers/hamilton-2016-diachronic-word-embeddings-reveal-statistical/) — William L. Hamilton, Jure Leskovec, Dan Jurafsky · 2016 · 1,270 citations
- [A Structural Probe for Finding Syntax in Word Representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/) — John Hewitt, Christopher D. Manning · 2019 · 1,228 citations
- [FastText.zip: Compressing text classification models](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/) — Armand Joulin, Edouard Grave, Piotr Bojanowski et al. · 2016 · 1,209 citations
- [Better Word Representations with Recursive Neural Networks for Morphology](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/) — Thang Luong, Richard Socher, Christopher Manning · 2013 · 1,017 citations
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://md.research.elamur.ai/free/en/papers/artetxe-2018-vecmap-unsupervised/) — Mikel Artetxe, Gorka Labaka, Eneko Agirre · 2018 · 847 citations
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/) — Wei Zhao, Maxime Peyrard, Fei Liu et al. · 2019 · 822 citations
- [Improving Vector Space Word Representations Using Multilingual Correlation](https://md.research.elamur.ai/free/en/papers/faruqui-2014-improving-vector-space-word-representations/) — Manaal Faruqui, Chris Dyer · 2014 · 819 citations
- [Linguistic Regularities in Sparse and Explicit Word Representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/) — Omer Levy, Yoav Goldberg · 2014 · 790 citations
- [Learning bilingual word embeddings with (almost) no bilingual data](https://md.research.elamur.ai/free/en/papers/artetxe-2017-learning-bilingual-word-embeddings-almost/) — Mikel Artetxe, Gorka Labaka, Eneko Agirre · 2017 · 681 citations
- [Bilingual Word Embeddings for Phrase-Based Machine Translation](https://md.research.elamur.ai/free/en/papers/zou-2013-bilingual-word-embeddings-phrase-based/) — Will Y. Zou, Richard Socher, Daniel Cer et al. · 2013 · 680 citations
- [Evaluation methods for unsupervised word embeddings](https://md.research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/) — Tobias Schnabel, Igor Labutov, David Mimno et al. · 2015 · 669 citations
- [Learning Word Vectors for 157 Languages](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/) — Edouard Grave, Piotr Bojanowski, Prakhar Gupta et al. · 2018 · 640 citations
- [Advances in Pre-Training Distributed Word Representations](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/) — Tomas Mikolov, Edouard Grave, Piotr Bojanowski et al. · 2017 · 614 citations
- [Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings](https://md.research.elamur.ai/free/en/papers/liu-2015-fine-grained-opinion-mining-recurrent/) — Pengfei Liu, Shafiq Joty, Helen Meng · 2015 · 589 citations
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/) — Matthew E. Peters, Mark Neumann, Luke Zettlemoyer et al. · 2018 · 585 citations
- [Measuring Bias in Contextualized Word Representations](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/) — Keita Kurita, Nidhi Vyas, Ayush Pareek et al. · 2019 · 584 citations
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/) — Ian Tenney, Patrick Xia, Berlin Chen et al. · 2019 · 582 citations
- [Learning principled bilingual mappings of word embeddings while preserving monolingual invariance](https://md.research.elamur.ai/free/en/papers/artetxe-2016-learning-principled-bilingual-mappings-word/) — Mikel Artetxe, Gorka Labaka, Eneko Agirre · 2016 · 533 citations
- [Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation](https://md.research.elamur.ai/free/en/papers/xing-2015-normalized-word-embedding-orthogonal-transform/) — Chao Xing, Dong Wang, Chao Liu et al. · 2015 · 528 citations

## Related concepts

- [Distributional semantics](https://md.research.elamur.ai/free/en/concepts/distributional-semantics/) — Distributional semantics is the study and modelling of meaning through the statistics of linguistic context. Words, phrases or sentences…
- [Semantic similarity](https://md.research.elamur.ai/free/en/concepts/semantic-similarity/) — Semantic similarity is a graded measure of how close two linguistic units, such as words, concepts, sentences or documents, are in meaning.…
- [Language model](https://md.research.elamur.ai/free/en/concepts/language-model/) — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- [Transformer and attention](https://md.research.elamur.ai/free/en/concepts/transformer/) — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…

## More papers

Topics: [📖 Lexical semantics & word embeddings](https://md.research.elamur.ai/free/en/topics/lexical-semantics/) · [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

Shelves: [💬 Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

- [← previous concept Universal Dependencies](https://md.research.elamur.ai/free/en/concepts/universal-dependencies/)
- [next concept → Word sense disambiguation](https://md.research.elamur.ai/free/en/concepts/word-sense-disambiguation/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)