---
title: "Tying Word Vectors and Word Classifiers: A Loss Framework for"
description: "Standard recurrent language models treat next-word prediction as ordinary classification against… 27 references, cited by 126 papers here."
authors: ["Hakan Inan", "Khashayar Khosravi", "Richard Socher"]
year: 2016
venue: "arXiv:1611.01462"
source: "https://arxiv.org/abs/1611.01462"
doi: "10.48550/arxiv.1611.01462"
arxiv: "1611.01462"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

# Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

**Authors:** [Hakan Inan](https://md.research.elamur.ai/free/en/authors/hakan-inan/), Khashayar Khosravi and [Richard Socher](https://md.research.elamur.ai/free/en/authors/richard-socher/) · **Year:** 2016 · **Venue:** arXiv:1611.01462 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1611.01462) · [PDF](https://arxiv.org/pdf/1611.01462) · [DOI 10.48550/arxiv.1611.01462](https://doi.org/10.48550/arxiv.1611.01462) · [arXiv 1611.01462](https://arxiv.org/abs/1611.01462) · [OpenAlex](https://openalex.org/W2549416390)

**Citations:** 288 citations (OpenAlex, 2026-10-14) · [126](https://md.research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard recurrent language models treat next-word prediction as ordinary classification against one-hot targets. Two consequences follow. The model ignores the fact that words already form a meaningful similarity space. It also learns separate input and output word representations, even though both describe the same vocabulary. Inan, Khosravi and Socher address this with a new training objective. Alongside the usual cross-entropy, they add a KL-divergence term that pulls the model's temperature-scaled prediction toward a soft target. That target is built from embedding similarity between the gold word and every other word. A theoretical analysis, under idealized conditions such as high temperature, zero training loss and equal embedding and hidden sizes, shows that this term pushes the output projection into the subspace spanned by the input embeddings. This motivates explicitly reusing the embedding matrix, transposed, as the output classifier, with no output bias. Experiments with two-layer LSTMs on Penn Treebank and Wikitext-2 report that this beats conventional training. The authors claim state-of-the-art Penn Treebank results while sharply cutting parameter counts.

## Key points
- **Augmented loss (AL):** the total loss is cross-entropy plus α times KL(ỹ ‖ ŷ). The soft target ỹ is a softmax over inner products between the target word's embedding and all embeddings, divided by a temperature τ.
- **Weight tying (reuse of embeddings):** the output matrix W is set equal to Lᵀ, removing one vocabulary-sized matrix. For large networks, most of the gain came from tying alone.
- **Setup:**
  - Two-layer LSTMs in small, medium and large sizes.
  - Backpropagation unrolled for 35 steps.
  - Gal (2015)-style variational dropout. The PTB models used rates of 0.7 (small), 0.5 (medium) and 0.35 (large), while the Wikitext-2 models used 0.8 (small) and 0.6 (medium).
- **AL hyperparameters:** τ = 20 and α = γτ, with γ of 0.5–0.8 for PTB and 1.0–1.5 for Wikitext-2. Results were robust to moderate changes in either value.
- **Analysis tool:** the appendix defines a subspace distance, the average squared sine of the principal angles, used to compare the spans of W and Lᵀ. The supplied excerpt omits the actual perplexity figures.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Hakan Inan, Khashayar Khosravi and Richard Socher. 2016. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. arXiv:1611.01462.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{inan2016tying,
  title = {{Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling}},
  author = {Hakan Inan, Khashayar Khosravi and Richard Socher},
  year = {2016},
  doi = {10.48550/arxiv.1611.01462},
  eprint = {1611.01462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.01462}
}
```

- [← previous Improving Coreference Resolution by Learning Entity-Level Distributed Representations](https://md.research.elamur.ai/free/en/papers/clark-2016-improving-coreference-resolution-learning-entity/)
- [next → A Character-level Decoder without Explicit Segmentation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

27 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). 2001. URL [http://www.iro.umontreal.ca/˜lisa/pointeurs/nips00_lm.ps](http://www.iro.umontreal.ca/˜lisa/pointeurs/nips00_lm.ps). [↗](http://www.iro.umontreal.ca/˜lisa/pointeurs/nips00_lm.ps)
2. ◦ Ake Björck and Gene H Golub. Numerical methods for computing angles between linear subspaces. Mathematics of computation, 27(123):579–594, 1973. [find](https://scholar.google.com/scholar?q=%E2%97%A6+Ake+Bjo%CC%88rck+and+Gene+H+Golub%2E+Numerical+methods+for+computing+angles+between+linear+subspaces%2E+Mathematics+of+computation%2C+27%28123%29%3A579%E2%80%93594%2C+1973%2E)
3. Wei-Chen Cheng, Stanley Kok, Hoai Vu Pham, Hai Leong Chieu, and Kian Ming Adam Chai. Language modeling with sum-product networks. 2014. [find](https://scholar.google.com/scholar?q=Wei%2DChen+Cheng%2C+Stanley+Kok%2C+Hoai+Vu+Pham%2C+Hai+Leong+Chieu%2C+and+Kian+Ming+Adam+Chai%2E+Language+modeling+with+sum%2Dproduct+networks%2E+2014%2E)
4. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. [On the properties of neural machine translation: Encoder-decoder approaches](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/). arXiv preprint arXiv:1409.1259, 2014.
5. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. [Multi-way, multilingual neural machine translation with a shared attention mechanism](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/). arXiv preprint arXiv:1601.01073, 2016.
6. Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with a wasserstein loss. In Advances in Neural Information Processing Systems, pp. 2053– 2061, 2015. [find](https://scholar.google.com/scholar?q=Charlie+Frogner%2C+Chiyuan+Zhang%2C+Hossein+Mobahi%2C+Mauricio+Araya%2C+and+Tomaso+A+Poggio%2E+Learning+with+a+wasserstein+loss%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2053%E2%80%93+2061%2C+2015%2E)
7. Yarin Gal. [A theoretically grounded application of dropout in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/). arXiv preprint arXiv:1512.05287, 2015.
8. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531, 2015.
9. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
10. Hakan Inan and Khashayar Khosravi. Improved learning through augmenting the loss. Stanford CS 224D: Deep Learning for Natural Language Processing, Spring 2016, 2016. [find](https://scholar.google.com/scholar?q=Hakan+Inan+and+Khashayar+Khosravi%2E+Improved+learning+through+augmenting+the+loss%2E+Stanford+CS+224D%3A+Deep+Learning+for+Natural+Language+Processing%2C+Spring+2016%2C+2016%2E)
11. Kazuki Irie, Zoltán Tüske, Tamer Alkhouli, Ralf Schlüter, and Hermann Ney. Lstm, gru, highway and a bit of attention: an empirical overview for language modeling in speech recognition. Interspeech, San Francisco, CA, USA, 2016. [find](https://scholar.google.com/scholar?q=Kazuki+Irie%2C+Zolta%CC%81n+Tu%CC%88ske%2C+Tamer+Alkhouli%2C+Ralf+Schlu%CC%88ter%2C+and+Hermann+Ney%2E+Lstm%2C+gru%2C+highway+and+a+bit+of+attention%3A+an+empirical+overview+for+language+modeling+in+speech+recognition%2E+Interspeech%2C)
12. Camille Jordan. Essai sur la géométrie à n dimensions. Bulletin de la Société mathématique de France, 3:103–174, 1875. [find](https://scholar.google.com/scholar?q=Camille+Jordan%2E+Essai+sur+la+ge%CC%81ome%CC%81trie+a%CC%80+n+dimensions%2E+Bulletin+de+la+Socie%CC%81te%CC%81+mathe%CC%81matique+de+France%2C+3%3A103%E2%80%93174%2C+1875%2E)
13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). arXiv preprint arXiv:1508.06615, 2015.
14. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 19(2):313–330, 1993.
15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843, 2016.
16. Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov+and+Geoffrey+Zweig%2E+Context+dependent+recurrent+neural+network+language+model%2E+Tomas+Mikolov%2C+Martin+Karafia%CC%81t%2C+Lukas+Burget%2C+Jan+Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+Recurrent+neural+netwo)
17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in neural information processing systems, pp. 3111–3119, 2013.
18. Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, pp. 641–648. ACM, 2007. [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Geoffrey+Hinton%2E+Three+new+graphical+models+for+statistical+language+modelling%2E+In+Proceedings+of+the+24th+international+conference+on+Machine+learning%2C+pp%2E+641%E2%80%93648%2E+ACM%2C+2007%2E)
19. Ramesh Nallapati, Bowen Zhou, Çaglar Gulçehre, and Bing Xiang. [Abstractive text summarization using sequence-to-sequence rnns and beyond](https://md.research.elamur.ai/free/en/papers/nallapati-2016-abstractive-text-summarization-sequence-sequence/). 2016.
20. Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. [How to construct deep recurrent neural networks](https://md.research.elamur.ai/free/en/papers/pascanu-2013-construct-deep-recurrent-neural-networks/). CoRR, abs/1312.6026, 2013a.
21. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. ICML (3), 28:1310–1318, 2013b. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+ICML+%283%29%2C+28%3A1310%E2%80%931318%2C+2013b%2E)
22. Jeffrey Pennington, Richard Socher, and Christopher D Manning. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In EMNLP, volume 14, pp. 1532–43, 2014.
23. Ofir Press and Lior Wolf. [Using the output embedding to improve language models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). arXiv preprint arXiv:1608.05859, 2016.
24. Alexander M Rush, Sumit Chopra, and Jason Weston. [A neural attention model for abstractive sentence summarization](https://md.research.elamur.ai/free/en/papers/rush-2015-neural-attention-model-abstractive-sentence/). arXiv preprint arXiv:1509.00685, 2015.
25. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). Citeseer, 2013.
26. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). arXiv preprint arXiv:1409.2329, 2014.
27. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. [Recurrent highway networks](https://md.research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/). arXiv preprint arXiv:1607.03474, 2016.

*Updated 2 November 2026*