---
title: "A Fast and Simple Algorithm for Training Neural Probabilistic"
description: "Neural probabilistic language models (NPLMs) tend to beat n-gram models, but they are slow to train. 22 references, cited by 113 papers here."
authors: ["Andriy Mnih", "Yee Whye Teh"]
year: 2012
venue: "arXiv:1206.6426 · In Proceedings of the 29th International Conference on Machine Learning, pages 1751-1758, 2012"
source: "https://arxiv.org/abs/1206.6426"
doi: "10.48550/arxiv.1206.6426"
arxiv: "1206.6426"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mnih-2012-fast-simple-algorithm-training-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mnih-2012-fast-simple-algorithm-training-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

# A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

**Authors:** [Andriy Mnih](https://md.research.elamur.ai/free/en/authors/andriy-mnih/) and [Yee Whye Teh](https://md.research.elamur.ai/free/en/authors/yee-whye-teh/) · **Year:** 2012 · **Venue:** arXiv:1206.6426 · In Proceedings of the 29th International Conference on Machine Learning, pages 1751-1758, 2012 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1206.6426) · [PDF](https://arxiv.org/pdf/1206.6426) · [DOI 10.48550/arxiv.1206.6426](https://doi.org/10.48550/arxiv.1206.6426) · [arXiv 1206.6426](https://arxiv.org/abs/1206.6426) · [OpenAlex](https://openalex.org/W2120861206)

**Citations:** 295 citations (OpenAlex, 2026-10-14) · [113](https://md.research.elamur.ai/free/en/papers/mnih-2012-fast-simple-algorithm-training-neural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural probabilistic language models (NPLMs) tend to beat n-gram models, but they are slow to train. The softmax normalization forces every gradient step to score the entire vocabulary. Mnih and Teh adapt noise-contrastive estimation (NCE), introduced by Gutmann and Hyvärinen for unnormalized densities, to discrete word distributions. NCE turns learning into a binary classification task: the model learns to tell observed next words apart from samples drawn from a simple unigram noise distribution, and normalizing constants are handled as parameters rather than computed explicitly. As the number of noise samples grows, the gradient approaches the maximum likelihood gradient.

The method is tested with a log-bilinear (LBL) model. On Penn Treebank it cuts training time by more than tenfold with no loss in model quality, and it is more stable than importance sampling while needing fewer samples. Trained at scale on Project Gutenberg texts, the LBL models reach a new best accuracy on the Microsoft Research Sentence Completion Challenge. This makes neural language models practical for large corpora.

## Key points
- NCE uses a fixed, context-independent unigram noise distribution. Unlike importance sampling (Bengio & Senécal), it needs no adaptive proposal and no growing sample count to stay stable.
- On Penn Treebank, NCE-trained LBL models match maximum-likelihood quality in under one-tenth of the training time.
- The large-scale corpus comes from 522 Project Gutenberg works, tokenized with NLTK (Punkt sentence tokenizer and Penn Treebank word tokenizer). It yields a 47M-word training set and a vocabulary of just under 80K words. Each model trained in 1–2 days on a single CPU core.
- On all 1,040 MSR Sentence Completion sentences, the results compare as follows:
  - An LBL model with a 10-word context and 300-dimensional features scored 54.7% correct.
  - The previous LSA record was 49%.
  - The best Kneser-Ney n-gram baseline (SRILM) scored 39.1%.
- Bidirectional-context LBL models had much lower perplexity but were noticeably worse at sentence completion than unidirectional ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 107 works · 22,924 citations

Also on the byline: [Gatsby Computational Neuroscience Unit](https://ror.org/0269epf40).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Andriy Mnih and Yee Whye Teh. 2012. A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. arXiv:1206.6426 · In Proceedings of the 29th International Conference on Machine Learning, pages 1751-1758, 2012.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mnih2012fast,
  title = {{A Fast and Simple Algorithm for Training Neural Probabilistic Language Models}},
  author = {Andriy Mnih and Yee Whye Teh},
  year = {2012},
  doi = {10.48550/arxiv.1206.6426},
  eprint = {1206.6426},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1206.6426}
}
```

- [← previous PATTY: A Taxonomy of Relational Patterns with Semantic Types](https://md.research.elamur.ai/free/en/papers/nakashole-2012-patty-taxonomy-relational-patterns-semantic/)
- [next → Sentence Simplification by Monolingual Machine Translation](https://md.research.elamur.ai/free/en/papers/wubben-2012-sentence-simplification-monolingual-machine-translation/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bengio, Yoshua and Senécal, Jean-Sébastien. Quick training of probabilistic neural nets by importance sampling. In AISTATS’03, 2003. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua+and+Sene%CC%81cal%2C+Jean%2DSe%CC%81bastien%2E+Quick+training+of+probabilistic+neural+nets+by+importance+sampling%2E+In+AISTATS%E2%80%9903%2C+2003%2E)
2. Bengio, Yoshua and Senécal, Jean-Sébastien. Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Transactions on Neural Networks, 19(4):713–722, 2008. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua+and+Sene%CC%81cal%2C+Jean%2DSe%CC%81bastien%2E+Adaptive+importance+sampling+to+accelerate+training+of+a+neural+probabilistic+language+model%2E+IEEE+Transactions+on+Neural+Networks%2C+19%284%29%3A713%E2%80%93722%2C+2008%2E)
3. Bengio, Yoshua, Ducharme, Réjean, and Vincent, Pascal. A neural probabilistic language model. In NIPS, pp. 932–938, 2000. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua%2C+Ducharme%2C+Re%CC%81jean%2C+and+Vincent%2C+Pascal%2E+A+neural+probabilistic+language+model%2E+In+NIPS%2C+pp%2E+932%E2%80%93938%2C+2000%2E)
4. Bengio, Yoshua, Ducharme, Rejean, Vincent, Pascal, and Jauvin, Christian. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). Journal of Machine Learning Research, 3:1137–1155, 2003.
5. Berger, A.L., Pietra, V.J.D., and Pietra, S.A.D. [A maximum entropy approach to natural language processing](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/). Computational linguistics, 22(1):39–71, 1996.
6. Bird, S., Klein, E., and Loper, E. Natural language processing with Python. O’Reilly Media, 2009. [find](https://scholar.google.com/scholar?q=Bird%2C+S%2E%2C+Klein%2C+E%2E%2C+and+Loper%2C+E%2E+Natural+language+processing+with+Python%2E+O%E2%80%99Reilly+Media%2C+2009%2E)
7. Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, 2008. [find](https://scholar.google.com/scholar?q=Collobert%2C+R%2E+and+Weston%2C+J%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+International+Conference+on+Machine+Learni)
8. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS2010), 2010. [find](https://scholar.google.com/scholar?q=Gutmann%2C+M%2E+and+Hyva%CC%88rinen%2C+A%2E+Noise%2Dcontrastive+estimation%3A+A+new+estimation+principle+for+unnormalized+statistical+models%2E+In+Proc%2E+Int%2E+Conf%2E+on+Artificial+Intelligence+and+Statistics+%28AISTATS2010%29)
9. Gutmann, M.U. and Hyvärinen, A. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13:307–361, 2012. [find](https://scholar.google.com/scholar?q=Gutmann%2C+M%2EU%2E+and+Hyva%CC%88rinen%2C+A%2E+Noise%2Dcontrastive+estimation+of+unnormalized+statistical+models%2C+with+applications+to+natural+image+statistics%2E+Journal+of+Machine+Learning+Research%2C+13%3A307%E2%80%93361%2C+2012%2E)
10. Maas, A.L. and Ng, A.Y. A probabilistic model for semantic word vectors. In NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning, 2010. [find](https://scholar.google.com/scholar?q=Maas%2C+A%2EL%2E+and+Ng%2C+A%2EY%2E+A+probabilistic+model+for+semantic+word+vectors%2E+In+NIPS+2010+Workshop+on+Deep+Learning+and+Unsupervised+Feature+Learning%2C+2010%2E)
11. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Karafia%CC%81t%2C+M%2E%2C+Burget%2C+L%2E%2C+C%CC%8Cernocky%CC%80%2C+J%2E%2C+and+Khudanpur%2C+S%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+Annual+Conference+of+the+International+Speech+Communication+Associa)
12. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Černockỳ, J. Empirical evaluation and combination of advanced language modeling techniques. In Proceedings of Interspeech, pp. 605–608, 2011. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Deoras%2C+A%2E%2C+Kombrink%2C+S%2E%2C+Burget%2C+L%2E%2C+and+C%CC%8Cernocky%CC%80%2C+J%2E+Empirical+evaluation+and+combination+of+advanced+language+modeling+techniques%2E+In+Proceedings+of+Interspeech%2C+pp%2E+605%E2%80%93608%2C+2011%2E)
13. Mnih, A. and Hinton, G. Three new graphical models for statistical language modelling. Proceedings of the 24th International Conference on Machine Learning, pp. 641–648, 2007. [find](https://scholar.google.com/scholar?q=Mnih%2C+A%2E+and+Hinton%2C+G%2E+Three+new+graphical+models+for+statistical+language+modelling%2E+Proceedings+of+the+24th+International+Conference+on+Machine+Learning%2C+pp%2E+641%E2%80%93648%2C+2007%2E)
14. Mnih, A., Yuecheng, Z., and Hinton, G. Improving a statistical language model through non-linear prediction. Neurocomputing, 72(7-9):1414–1418, 2009. [find](https://scholar.google.com/scholar?q=Mnih%2C+A%2E%2C+Yuecheng%2C+Z%2E%2C+and+Hinton%2C+G%2E+Improving+a+statistical+language+model+through+non%2Dlinear+prediction%2E+Neurocomputing%2C+72%287%2D9%29%3A1414%E2%80%931418%2C+2009%2E)
15. Mnih, Andriy and Hinton, Geoffrey. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, volume 21, 2009. [find](https://scholar.google.com/scholar?q=Mnih%2C+Andriy+and+Hinton%2C+Geoffrey%2E+A+scalable+hierarchical+distributed+language+model%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+21%2C+2009%2E)
16. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In AISTATS’05, pp. 246–252, 2005. [find](https://scholar.google.com/scholar?q=Morin%2C+Frederic+and+Bengio%2C+Yoshua%2E+Hierarchical+probabilistic+neural+network+language+model%2E+In+AISTATS%E2%80%9905%2C+pp%2E+246%E2%80%93252%2C+2005%2E)
17. Pihlaja, M., Gutmann, M., and Hyvärinen, A. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI2010)., 2010. [find](https://scholar.google.com/scholar?q=Pihlaja%2C+M%2E%2C+Gutmann%2C+M%2E%2C+and+Hyva%CC%88rinen%2C+A%2E+A+family+of+computationally+efficient+and+simple+estimators+for+unnormalized+statistical+models%2E+In+Proc%2E+Conf%2E+on+Uncertainty+in+Artificial+Intelligence+%28)
18. Schwenk, Holger and Gauvain, Jean-Luc. Training neural network language models on very large corpora. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 201–208, 2005. [find](https://scholar.google.com/scholar?q=Schwenk%2C+Holger+and+Gauvain%2C+Jean%2DLuc%2E+Training+neural+network+language+models+on+very+large+corpora%2E+In+Proceedings+of+Human+Language+Technology+Conference+and+Conference+on+Empirical+Methods+in+Natu)
19. Socher, R., Lin, C.C., Ng, A.Y., and Manning, C.D. Parsing natural scenes and natural language with recursive neural networks. In International Conference on Machine Learning (ICML), 2011. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Lin%2C+C%2EC%2E%2C+Ng%2C+A%2EY%2E%2C+and+Manning%2C+C%2ED%2E+Parsing+natural+scenes+and+natural+language+with+recursive+neural+networks%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2011%2E)
20. Stolcke, A. SRILM – an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pp. 901–904, 2002. [find](https://scholar.google.com/scholar?q=Stolcke%2C+A%2E+SRILM+%E2%80%93+an+extensible+language+modeling+toolkit%2E+In+Proceedings+of+the+International+Conference+on+Spoken+Language+Processing%2C+volume+2%2C+pp%2E+901%E2%80%93904%2C+2002%2E)
21. Turian, J., Ratinov, L., and Bengio, Y. [Word representations: A simple and general method for semi-supervised learning](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/). In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384–394, 2010.
22. Zweig, G. and Burges, C.J.C. The Microsoft Research Sentence Completion Challenge. Technical Report MSR-TR-2011-129, Microsoft Research, 2011. [find](https://scholar.google.com/scholar?q=Zweig%2C+G%2E+and+Burges%2C+C%2EJ%2EC%2E+The+Microsoft+Research+Sentence+Completion+Challenge%2E+Technical+Report+MSR%2DTR%2D2011%2D129%2C+Microsoft+Research%2C+2011%2E)

*Updated 2 November 2026*