---
title: "Directional Skip-Gram — paper by Song et al. (2018)"
description: "The standard skip-gram (SG) model learns word vectors from co-occurrence alone and does not… 19 references, cited by 62 papers here."
authors: ["Yan Song", "Shuming Shi", "Jing Li", "Haisong Zhang"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)"
source: "https://aclanthology.org/N18-2028/"
doi: "10.18653/v1/n18-2028"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/song-2018-directional-skip-gram-explicitly-distinguishing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2018-directional-skip-gram-explicitly-distinguishing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings

# Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings

**Authors:** [Yan Song](https://md.research.elamur.ai/free/en/authors/yan-song/), [Shuming Shi](https://md.research.elamur.ai/free/en/authors/shuming-shi/), [Jing Li](https://md.research.elamur.ai/free/en/authors/jing-li/) and [Haisong Zhang](https://md.research.elamur.ai/free/en/authors/haisong-zhang/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2018](https://md.research.elamur.ai/free/en/venues/naacl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-2028/) · [PDF](https://aclanthology.org/N18-2028.pdf) · [DOI 10.18653/v1/n18-2028](https://doi.org/10.18653/v1/n18-2028) · [OpenAlex](https://openalex.org/W2804552794)

**Citations:** 380 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/song-2018-directional-skip-gram-explicitly-distinguishing/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The standard skip-gram (SG) model learns word vectors from co-occurrence alone and does not register whether a neighbour appears before or after the target word. Earlier order-aware extensions, such as the structured skip-gram (SSG), keep a separate output vector for each relative position. This inflates memory and training time and makes them vulnerable to sparse data. Song et al. (Tencent AI Lab) propose directional skip-gram (DSG). It gives every word one extra "direction" vector, which is trained with a sigmoid objective to tell whether a context word lies to the left or to the right. This direction score is added to the usual SG prediction. The extra cost is one operation per word, so training speed remains close to SG.

Embeddings were trained on an English Wikipedia dump of roughly 2 billion tokens. They were evaluated on word similarity and on POS tagging, the latter with a BiLSTM-CRF. DSG outperformed SG, SSG and a simplified SSG (SSSG) in both settings, and it held up especially well on a tiny training corpus. The result shows that a cheap, binary notion of word order can improve embeddings without extra resources.

## Key points
- **Cost:** DSG and SSSG both need 3|V|d parameters, compared with 2|V|d for SG and (2c+1)|V|d for SSG. Operations scale as 2cC(n+2)o for DSG versus 4c²C(n+1)o for SSG, so DSG's speed gap to SG narrows as the context window widens.
- **Training setup:** all models shared the same hyperparameters, namely 200 dimensions, window 5, 5 negative samples, 5 iterations and a starting learning rate of 0.025.
- **Word similarity, full Wiki corpus:** DSG was the best of the SG-family models, with Spearman ρ×100 of 73.76 on MEN-3k, 36.10 on SimLex-999 and 72.60 on WS-353. CWin scored higher only on MEN-3k (74.28).
- **Word similarity, 0.1% Wiki sample:** this sample simulated a cold-start setting, and DSG was best on all three datasets (63.18, 27.51 and 66.71). SSG and CWin degraded markedly on the same sample.
- **POS tagging, test accuracy:** on WSJ (Penn Treebank) DSG reached 97.37, ahead of SSG at 97.31 and SG at 97.12. On the ARK tweets it reached 90.43, ahead of SSG at 90.15 and SG at 88.77; the ARK embeddings came from a 100-million-tweet TWT corpus.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 66 works · 17,945 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Yan Song, Shuming Shi, Jing Li and Haisong Zhang. 2018. Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{song2018directional,
  title = {{Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings}},
  author = {Yan Song, Shuming Shi, Jing Li and Haisong Zhang},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2028},
  url = {https://doi.org/10.18653/v1/n18-2028}
}
```

- [← previous Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/)
- [next → Deep Communicating Agents for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2044–2048. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Felix+Hill%2C+and+Stephen+Clark%2E+2015%2E+Specializing+word+embeddings+for+similarity+or+relatedness%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
2. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. [Tailoring Continuous Word Representations for Dependency Parsing](https://md.research.elamur.ai/free/en/papers/bansal-2014-tailoring-continuous-word-representations-dependency/). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Baltimore, Maryland, pages 809–815.
3. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. [Neural Architectures for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/).
4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Re%CC%81jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Janvin%2E+2003%2E)
5. A Neural Probabilistic Lan-Yan Song and Chia-Jung Lee. 2017. Learning user embeddings from emails. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain, pages 733–738. [find](https://scholar.google.com/scholar?q=A+Neural+Probabilistic+Lan%2DYan+Song+and+Chia%2DJung+Lee%2E+2017%2E+Learning+user+embeddings+from+emails%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Li)
6. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, California, pages 260–270. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies%2E+San+Diego%2C+California%2C+pages+260%E2%80%93270%2E)
7. Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015a. [Two/Too Simple Adaptations of Word2Vec for Syntax Problems](https://md.research.elamur.ai/free/en/papers/ling-2015-two-too-simple-adaptations-word2vec/). In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computationa…
8. Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learning word representations with regularization from prior knowledge. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Vancouver, Canada, pages 143–152. [find](https://scholar.google.com/scholar?q=Yan+Song%2C+Chia%2DJung+Lee%2C+and+Fei+Xia%2E+2017%2E+Learning+word+representations+with+regularization+from+prior+knowledge%2E+In+Proceedings+of+the+21st+Conference+on+Computational+Natural+Language+Learning+%28Co)
9. Peter D. Turney and Patrick Pantel. 2010. [From Frequency to Meaning: Vector Space Models of Semantics](https://md.research.elamur.ai/free/en/papers/turney-2010-frequency-meaning-vector-space-models/). Journal of Artificial Intelligence Research 37(1):141–188.
10. Wang Ling, Yulia Tsvetkov, Silvio Amir, Ramon Fermandez, Chris Dyer, Alan W Black, Isabel Trancoso, and Chu-Cheng Lin. 2015b. Not All Contexts Are Created Equal: Better Word Representations with Variable Attention. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proces… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Yulia+Tsvetkov%2C+Silvio+Amir%2C+Ramon+Fermandez%2C+Chris+Dyer%2C+Alan+W+Black%2C+Isabel+Trancoso%2C+and+Chu%2DCheng+Lin%2E+2015b%2E+Not+All+Contexts+Are+Created+Equal%3A+Better+Word+Representations+with+Varia)
11. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv pre-print abs/1502.05698.
12. Mo Yu and Mark Dredze. 2014. [Improving lexical embeddings with semantic knowledge](https://md.research.elamur.ai/free/en/papers/yu-2014-improving-lexical-embeddings-semantic-knowledge/). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Baltimore, Maryland, pages 545–550.
13. Mitch Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. [Building a large annotated corpus of English: the Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics 19(2):313–330.
14. Jerrold H Zar. 1998. Spearman rank correlation. Encyclopedia of Biostatistics . [find](https://scholar.google.com/scholar?q=Jerrold+H+Zar%2E+1998%2E+Spearman+rank+correlation%2E+Encyclopedia+of+Biostatistics+%2E)
15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. [Efficient Estimation of Word Representations in Vector Space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint abs/1301.3781.
16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. [Distributed Representations of Words and Phrases and their Compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, page…
17. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. [Linguistic Regularities in Continuous Space Word Representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Li…
18. Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin, Germany, pa… [find](https://scholar.google.com/scholar?q=Kim+Anh+Nguyen%2C+Sabine+Schulte+im+Walde%2C+and+Ngoc+Thang+Vu%2E+2016%2E+Integrating+distributional+lexical+contrast+into+word+embeddings+for+antonym%2Dsynonym+distinction%2E+In+Proceedings+of+the+54th+Annual+Me)
19. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. [Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters](https://md.research.elamur.ai/free/en/papers/owoputi-2013-improved-part-speech-tagging-online/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguist…

*Updated 24 October 2026*