---
title: "Two/Too Simple Adaptations of Word2Vec for — Ling 2015"
description: "Word2Vec's skip-gram and CBOW models ignore the order of context words."
authors: ["Wang Ling", "Chris Dyer", "Alan W. Black", "Isabel Trancoso"]
year: 2015
venue: "Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N15-1142/"
doi: "10.3115/v1/n15-1142"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ling-2015-two-too-simple-adaptations-word2vec/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ling-2015-two-too-simple-adaptations-word2vec/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Two/Too Simple Adaptations of Word2Vec for Syntax Problems

# Two/Too Simple Adaptations of Word2Vec for Syntax Problems

**Authors:** [Wang Ling](https://md.research.elamur.ai/free/en/authors/wang-ling/), [Chris Dyer](https://md.research.elamur.ai/free/en/authors/chris-dyer/), [Alan W. Black](https://md.research.elamur.ai/free/en/authors/alan-w-black/), [Isabel Trancoso](https://md.research.elamur.ai/free/en/authors/isabel-trancoso/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N15-1142/) · [PDF](https://aclanthology.org/N15-1142.pdf) · [DOI 10.3115/v1/n15-1142](https://doi.org/10.3115/v1/n15-1142) · [OpenAlex](https://openalex.org/W2296194829)

**Citations:** 384 citations (openalex-snapshot · 2026-10-14) · [120](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Word2Vec's skip-gram and CBOW models ignore the order of context words. That works well for semantic similarity but is a poor fit for syntax, where a word's position relative to its neighbours carries much of the signal. The authors propose two lightweight, order-aware variants. Structured Skip-gram uses a separate output matrix for each relative context position. CWindow replaces CBOW's summed context with an ordered concatenation of context embeddings. They trained 50-dimensional vectors on a large 2014 English Wikipedia dump, WIKI(L), and on a large tweet sample, TWITTER, using a single day on ordinary hardware. They then used these vectors to initialise neural taggers and parsers. On Penn Treebank POS tagging, ARK Twitter POS tagging, and PTB dependency parsing with Stanford Dependencies, both variants beat the original models. They also matched or exceeded SENNA embeddings. Qualitatively, nearest neighbours grouped words more by syntactic class and captured spelling variants in tweets. The work matters because it gives syntax-friendly embeddings with little extra cost or complexity, and the code is released as wang2vec.

## Key points
- **Corpora:** WIKI(L) has about 1,897M words (60M sentences) and TWITTER has 56M tweets (847M words). WIKI(S), a 16M-word set, was used for contrast. Training used window c = 5, 10 negative samples, a minimum count of 40, and 50 dimensions.
- **POS tagging (test accuracy):**
  - On PTB, Structured Skip-gram reached 97.05 and CWindow 97.01, compared with 96.68 for Skip-gram, 96.13 for CBOW and 96.58 for SENNA.
  - On ARK Twitter, Structured Skip-gram reached 89.79, slightly above the 89.50 obtained with Brown clusters alone.
- **Dependency parsing (Chen and Manning parser, PTB test):** Structured Skip-gram scored UAS 92.24 / LAS 89.92. The other models scored:
  - Skip-gram: 91.90 / 89.55
  - CBOW: 91.52 / 88.93
  - SENNA: 92.03 / 89.51
- **Speed:** Structured Skip-gram processes 34.64k words/s versus 71.35k for Skip-gram. CWindow processes 124.43k words/s versus 342.17k for CBOW. Even so, every embedding set was trained in under a day.
- **Code:** released publicly at github.com/wlin12/wang2vec.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 114 not yet.

- [Diachronic Analysis of Phrasal Verbs in English Scientific Writing](https://md.research.elamur.ai/free/en/papers/alves-2025-diachronic-analysis-phrasal-verbs-english/)
- [Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
- [Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/marcheggiani-2017-encoding-sentences-graph-convolutional-networks/)
- [A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/marcheggiani-2017-simple-accurate-syntax-agnostic-neural/)
- [context2vec: Learning Generic Context Embedding with Bidirectional LSTM](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/)
- [Transition-Based Dependency Parsing with Stack Long Short-Term Memory](https://md.research.elamur.ai/free/en/papers/missing-2015-dyer-transition-based-dependency-parsing-stack/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ling2015two,
  title = {{Two/Too Simple Adaptations of Word2Vec for Syntax Problems}},
  author = {Ling, Wang and Dyer, Chris and Black, Alan W. and Trancoso, Isabel},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1142},
  url = {https://doi.org/10.3115/v1/n15-1142}
}
```

- [← previous Two Practical Rhetorical Structure Theory Parsers](https://md.research.elamur.ai/free/en/papers/surdeanu-2015-two-practical-rhetorical-structure-theory/)
- [next → U-Net: Convolutional Networks for Biomedical Image Segmentation](https://md.research.elamur.ai/free/en/papers/ronneberger-2015-u-net-convolutional-networks-biomedical/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas and Dan Klein. 2014. How much do word embeddings encode about syntax. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Jacob+Andreas+and+Dan+Klein%2E+2014%2E+How+much+do+word+embeddings+encode+about+syntax%2E+In+Proceedings+of+ACL%2E)
2. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mohit+Bansal%2C+Kevin+Gimpel%2C+and+Karen+Livescu%2E+2014%2E+Tailoring+continuous+word+representations+for+dependency+parsing%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguis) [our card](https://md.research.elamur.ai/free/en/papers/bansal-2014-tailoring-continuous-word-representations-dependency/)
3. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Christopher+D+Manning%2E+2014%2E+A+fast+and+accurate+dependency+parser+using+neural+networks%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNL) [our card](https://md.research.elamur.ai/free/en/papers/missing-2014-chen-fast-accurate-dependency-parser-neural/)
4. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493– 2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
5. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Rabih+Zbib%2C+Zhongqiang+Huang%2C+Thomas+Lamar%2C+Richard+Schwartz%2C+and+John+Makhoul%2E+2014%2E)
6. Fast and robust neural network joint models for statistical machine translation. In 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA, June. [find](https://scholar.google.com/scholar?q=Fast+and+robust+neural+network+joint+models+for+statistical+machine+translation%2E+In+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+Baltimore%2C+MD%2C+USA%2C+June%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/)
7. Paramveer Dhillon, Dean P Foster, and Lyle H Ungar. 2011. Multi-view learning of word embeddings via cca. In Advances in Neural Information Processing Systems, pages 199–207. [find](https://scholar.google.com/scholar?q=Paramveer+Dhillon%2C+Dean+P+Foster%2C+and+Lyle+H+Ungar%2E+2011%2E+Multi%2Dview+learning+of+word+embeddings+via+cca%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+199%E2%80%93207%2E)
8. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research, 11:625–660. [find](https://scholar.google.com/scholar?q=Dumitru+Erhan%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+Pierre%2DAntoine+Manzagol%2C+Pascal+Vincent%2C+and+Samy+Bengio%2E+2010%2E+Why+does+unsupervised+pre%2Dtraining+help+deep+learning%3F+The+Journal+of+Machine+Learning+Re)
9. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of EACL. [find](https://scholar.google.com/scholar?q=Manaal+Faruqui+and+Chris+Dyer%2E+2014%2E+Improving+vector+space+word+representations+using+multilingual+correlation%2E+In+Proceedings+of+EACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/faruqui-2014-improving-vector-space-word-representations/)
10. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Associ… [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Nathan+Schneider%2C+Brendan+O%E2%80%99Connor%2C+Dipanjan+Das%2C+Daniel+Mills%2C+Jacob+Eisenstein%2C+Michael+Heilman%2C+Dani+Yogatama%2C+Jeffrey+Flanigan%2C+and+Noah+A+Smith%2E+2011%2E+Part%2Dof%2Dspeech+tagging+for+twi) [our card](https://md.research.elamur.ai/free/en/papers/gimpel-2011-part-speech-tagging-twitter-annotation/)
11. Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. — [arXiv:1402.3722](https://arxiv.org/abs/1402.3722) [find](https://scholar.google.com/scholar?q=Yoav+Goldberg+and+Omer+Levy%2E+2014%2E+word2vec+explained%3A+deriving+mikolov+et+al%2E%E2%80%99s+negative%2Dsampling+word%2Dembedding+method%2E+arXiv+preprint+arXiv%3A1402%2E3722%2E+%E2%80%94+%5BarXiv%3A1402%2E3722%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1402)
12. Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Revisiting embedding features for simple semi-supervised learning. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jiang+Guo%2C+Wanxiang+Che%2C+Haifeng+Wang%2C+and+Ting+Liu%2E+2014%2E+Revisiting+embedding+features+for+simple+semi%2Dsupervised+learning%2E+In+Proceedings+of+EMNLP%2E)
13. Geoffrey E Hinton and Ruslan Salakhutdinov. 2012. A better way to pretrain deep boltzmann machines. In Advances in Neural Information Processing Systems, pages 2447–2455. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton+and+Ruslan+Salakhutdinov%2E+2012%2E+A+better+way+to+pretrain+deep+boltzmann+machines%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2447%E2%80%932455%2E)
14. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for… [find](https://scholar.google.com/scholar?q=Eric+H+Huang%2C+Richard+Socher%2C+Christopher+D+Manning%2C+and+Andrew+Y+Ng%2E+2012%2E+Improving+word+representations+via+global+context+and+multiple+word+prototypes%2E+In+Proceedings+of+the+50th+Annual+Meeting+of) [our card](https://md.research.elamur.ai/free/en/papers/huang-2012-improving-word-representations-global-context/)
15. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+EMNLP%2C+pages+1700%E2%80%931709%2E) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
16. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A. Smith. 2014. A dependency parser for tweets. In Proc. of EMNLP, pages 1001–1012, Doha, Qatar, October. [find](https://scholar.google.com/scholar?q=Lingpeng+Kong%2C+Nathan+Schneider%2C+Swabha+Swayamdipta%2C+Archna+Bhatia%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2014%2E+A+dependency+parser+for+tweets%2E+In+Proc%2E+of+EMNLP%2C+pages+1001%E2%80%931012%2C+Doha%2C+Qatar%2C+October%2E) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2014-kong-dependency-parser-tweets/)
17. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 2. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Dependency%2Dbased+word+embeddings%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+volume+2%2E)
18. Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. 2014. A recursive recurrent neural network for statistical machine translation. In Proceedings of ACL, pages 1491– 1500. [find](https://scholar.google.com/scholar?q=Shujie+Liu%2C+Nan+Yang%2C+Mu+Li%2C+and+Ming+Zhou%2E+2014%2E+A+recursive+recurrent+neural+network+for+statistical+machine+translation%2E+In+Proceedings+of+ACL%2C+pages+1491%E2%80%93+1500%2E)
19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Processing+)
20. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In HLT-NAACL, pages 380–390. [find](https://scholar.google.com/scholar?q=Olutobi+Owoputi%2C+Brendan+O%E2%80%99Connor%2C+Chris+Dyer%2C+Kevin+Gimpel%2C+Nathan+Schneider%2C+and+Noah+A+Smith%2E+2013%2E+Improved+part%2Dof%2Dspeech+tagging+for+online+conversational+text+with+word+clusters%2E+In+HLT%2DNAACL%2C+) [our card](https://md.research.elamur.ai/free/en/papers/owoputi-2013-improved-part-speech-tagging-online/)
21. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+VV+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+3104%E2%80%933112%2E)
22. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 384–394, Stroudsburg, PA, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Joseph+Turian%2C+Lev+Ratinov%2C+and+Yoshua+Bengio%2E+2010%2E+Word+representations%3A+A+simple+and+general+method+for+semi%2Dsupervised+learning%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
23. Yi Yang and Jacob Eisenstein. 2015. Unsupervised multi-domain adaptation with feature embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi+Yang+and+Jacob+Eisenstein%2E+2015%2E+Unsupervised+multi%2Ddomain+adaptation+with+feature+embeddings%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation)