---
title: "The Role of Context Types and Dimensionality — Melamud 2016"
description: "Skip-gram embeddings are usually trained on fixed-size word windows."
authors: ["Oren Melamud", "David McClosky", "Siddharth Patwardhan", "Mohit Bansal"]
year: 2016
venue: "Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N16-1118/"
doi: "10.18653/v1/n16-1118"
arxiv: "1601.00893"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/melamud-2016-role-context-types-dimensionality-learning/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/melamud-2016-role-context-types-dimensionality-learning/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2016](https://md.research.elamur.ai/free/papers/2016/page/3/) / The Role of Context Types and Dimensionality in Learning Word Embeddings

# The Role of Context Types and Dimensionality in Learning Word Embeddings

**Authors:** [Oren Melamud](https://md.research.elamur.ai/free/authors/oren-melamud/), [David McClosky](https://md.research.elamur.ai/free/authors/david-mcclosky/), Siddharth Patwardhan, [Mohit Bansal](https://md.research.elamur.ai/free/authors/mohit-bansal/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N16-1118/) · [PDF](https://aclanthology.org/N16-1118.pdf) · [DOI 10.18653/v1/n16-1118](https://doi.org/10.18653/v1/n16-1118) · [arXiv 1601.00893](https://arxiv.org/abs/1601.00893) · [OpenAlex](https://openalex.org/W2225261319)

**Citations:** 103 citations (openalex-snapshot · 2026-10-14) · [41](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Skip-gram embeddings are usually trained on fixed-size word windows. This paper asks how the choice of context, together with vector size, affects results across many tasks. The authors trained all models on one shared English corpus of roughly 10B words, built from Wikipedia 2015, UMBC and Gigaword and parsed with Stanford CoreNLP. They compared three context types:
- window contexts of sizes 1, 5 and 10
- dependency contexts, trained with word2vecf
- a new weighted substitute-word variant, where language-model fillers act as weighted contexts

They evaluated these on word similarity and relatedness benchmarks and on four downstream tasks: dependency parsing, NER, coreference and sentiment analysis. They also tested combinations, either by concatenating vectors or by merging them with SVD or CCA. On intrinsic benchmarks, particular context types and larger dimensionality were consistently preferred. Downstream tasks instead needed task-specific tuning of both settings. Once adding dimensions stopped helping, concatenating embeddings from different context types often brought further gains. The practical lesson is to choose embeddings for an applied system by testing them on that task, not by relying on intrinsic benchmark scores.

## Key points
- **Training setup:** Words seen fewer than 100 times were dropped, leaving a vocabulary of about 500K. Parses follow Universal Dependencies (Stanford CoreNLP 3.5.2), and the embeddings were released publicly.
- **Substitute-based embeddings (SUB):**
  - A 4-gram Kneser-Ney model built with KenLM, plus FASTSUBS, produced the substitute vectors.
  - Each vector was cut to its top-10 substitutes with normalized probabilities, and at most 20,000 vectors were generated per word type.
  - The word2vecf objective was changed so that each target–context pair counts in proportion to its weight.
- **Qualitative differences:** Large windows (W10) capture topical relatedness. Small windows and DEP favour functional similarity. SUB is the most functional and even tends to preserve verb tense.
- **Concatenation:**
  - It helps only once single embeddings are near saturation. For example, in sentiment at 50 dimensions, joining two 25-dimensional vectors did worse than one 50-dimensional vector.
  - The best combinations usually paired topical W10 with a functional type (SUB, DEP or W1).
- **SVD and CCA merging:** Both performed worse than single embeddings on every extrinsic task. This supports the view that different context types carry complementary information.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

41 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 38 not yet.

- [context2vec: Learning Generic Context Embedding with Bidirectional LSTM](https://md.research.elamur.ai/free/papers/melamud-2016-context2vec-learning-generic-context-embedding/)
- [Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network](https://md.research.elamur.ai/free/papers/mishra-2017-learning-cognitive-features-gaze-data/)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://md.research.elamur.ai/free/papers/peng-2019-transfer-learning-biomedical-natural-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{melamud2016role,
  title = {{The Role of Context Types and Dimensionality in Learning Word Embeddings}},
  author = {Melamud, Oren and McClosky, David and Patwardhan, Siddharth and Bansal, Mohit},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1118},
  eprint = {1601.00893},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n16-1118}
}
```

- [← previous The morphosyntax of aspect stacking in Northeastern Neo-Aramaic](https://md.research.elamur.ai/free/papers/lingbulk-2016-kalin-morphosyntax-aspect-stacking-northeastern-neo/)
- [next → The Role of Modifier and Head Properties in Predicting the Compositionality of English and German Noun-Noun Compounds: A Vector-Space Perspective](https://md.research.elamur.ai/free/papers/walde-2016-role-modifier-head-properties-predicting/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/papers/2016/page/3/)

## References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Agirre et al.2009] Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of NAACL. Association for Computational Linguistics. [Bansal et al.2014] Mohi… [find](https://scholar.google.com/scholar?q=%5BAgirre+et+al%2E2009%5D+Eneko+Agirre%2C+Enrique+Alfonseca%2C+Keith+Hall%2C+Jana+Kravalova%2C+Marius+Pas%CC%A7ca%2C+and+Aitor+Soroa%2E+2009%2E+A+study+on+similarity+and+relatedness+using+distributional+and+WordNet%2Dbased+appr)
2. [Goikoetxea et al.2016] Josu Goikoetxea, Eneko Agirre, and Aitor Soroa. 2016. Single or multiple? combining word representations independently learned from text and wordnet. In Proceedings of AAAI. [Han et al.2013] Lushan Han, Abhay L. Kashyap, Tim Finin, James Mayfield, and Johnathan Weese. 2013. U… [find](https://scholar.google.com/scholar?q=%5BGoikoetxea+et+al%2E2016%5D+Josu+Goikoetxea%2C+Eneko+Agirre%2C+and+Aitor+Soroa%2E+2016%2E+Single+or+multiple%3F+combining+word+representations+independently+learned+from+text+and+wordnet%2E+In+Proceedings+of+AAAI%2E+%5BH) [our card](https://md.research.elamur.ai/free/papers/hill-2015-simlex-999-evaluating-semantic-models/)
3. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. [Melamud et al.2014] Oren Melamud, Ido Dagan, Jacob Goldberger, Idan Szpektor, and… [find](https://scholar.google.com/scholar?q=Bethard%2C+and+David+McClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Proceedings+of+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+System+Demonstratio) [our card](https://md.research.elamur.ai/free/papers/manning-2014-stanford-corenlp-natural-language-processing/)