---
title: "A Neural Probabilistic Language Model — Bengio 2003"
description: "The authors propose a neural network model that simultaneously learns two things: a continuous feature vector for… 33 references, cited by 901 papers here."
authors: ["Yoshua Bengio", "Réjean Ducharme", "Pascal Vincent", "Christian Jauvin"]
year: 2003
venue: "JMLR 3:1137–1155"
source: "https://www.jmlr.org/papers/v3/bengio03a.html"
licence: "All rights reserved — free to read at the source"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2003](https://md.research.elamur.ai/free/en/papers/2003/) / A Neural Probabilistic Language Model

# A Neural Probabilistic Language Model

**Authors:** [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/), Réjean Ducharme, [Pascal Vincent](https://md.research.elamur.ai/free/en/authors/pascal-vincent/) and Christian Jauvin · **Year:** 2003 · **Venue:** JMLR 3:1137–1155 · **Licence:** All rights reserved — free to read at the source

**Links:** [Publisher's page](https://www.jmlr.org/papers/v3/bengio03a.html) · [PDF](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)

**Citations:** [901](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper addresses statistical language modeling, that is, estimating the probability of the next word given the preceding ones. The main obstacle here is the "curse of dimensionality": almost every test word sequence is absent from the training data. N-gram models generalize only by gluing together short fragments and do not take similarity between words into account. The authors propose a neural network model that simultaneously learns two things: a continuous feature vector for each word (distributed representation) and a probability function that depends on the vectors of the context words. Words with similar meaning or grammatical role receive similar vectors. As a result, a single training sentence increases the probability of a vast number of "neighboring" sentences. The model was evaluated on two corpora: Brown (over a million examples) and AP News (over 15 million words). It was compared against the best smoothed trigram models. The neural network achieved 10–20% lower perplexity and makes better use of longer context. The paper laid the foundation for word embeddings and neural language models, on which modern NLP rests.

## Key points
- **The scale problem:** to describe the joint distribution of 10 words with a vocabulary of 100,000 words, the n-gram approach potentially requires about 10^50 free parameters. A model with continuous vectors avoids this exponential growth: computation and memory grow linearly with the number of conditioning variables.
- **Architecture:** a word vector table C shared across all positions, with dimensionality m = 30, 60 or 100 (the vocabulary is about 17,000 words). The context vectors are fed into a multilayer neural network with a softmax output. The vectors and the network weights are trained jointly by maximizing a regularized log-likelihood.
- **Result:** on the Brown and AP News corpora, perplexity is 10–20% lower than that of a smoothed (interpolated / back-off) trigram. Mixing the neural network with a trigram improves the result even further.
- **Engineering contribution:** the authors showed that a model with millions of parameters can be trained on tens of millions of examples in reasonable time, in particular thanks to parallel computation. Later, Bengio and Senécal (2003) proposed a speedup of roughly 100 times via importance sampling.
- **Future directions:** a hierarchical (tree-structured) softmax with a gain of up to |V|/log|V|, recurrent or time-delay networks for longer context, multiple vectors for polysemous words, and the use of WordNet and parts of speech. Practical benefit for speech recognition has already been demonstrated by Schwenk and Gauvain (2002).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin. 2003. A Neural Probabilistic Language Model. JMLR 3:1137–1155.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bengio2003neural,
  title = {{A Neural Probabilistic Language Model}},
  author = {Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin},
  booktitle = {JMLR 3:1137–1155},
  year = {2003},
  url = {https://www.jmlr.org/papers/v3/bengio03a.html}
}
```

- [← previous Introduction to the Special Issue on the Web as Corpus](https://md.research.elamur.ai/free/en/papers/kilgarriff-2003-introduction-special-issue-web-as/)
- [next → Statistical Dependency Analysis with Support Vector Machines](https://md.research.elamur.ai/free/en/papers/yamada-2003-statistical-dependency-analysis-support-vector/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/en/papers/2003/)

## References

33 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Baker and A. McCallum. Distributional clustering of words for text classification. In SIGIR’98, 1998. [find](https://scholar.google.com/scholar?q=D%2E+Baker+and+A%2E+McCallum%2E+Distributional+clustering+of+words+for+text+classification%2E+In+SIGIR%E2%80%9998%2C+1998%2E)
2. J.R. Bellegarda. A latent semantic analysis framework for large–span language modeling. In Proceedings of Eurospeech 97, pages 1451–1454, Rhodes, Greece, 1997. [find](https://scholar.google.com/scholar?q=J%2ER%2E+Bellegarda%2E+A+latent+semantic+analysis+framework+for+large%E2%80%93span+language+modeling%2E+In+Proceedings+of+Eurospeech+97%2C+pages+1451%E2%80%931454%2C+Rhodes%2C+Greece%2C+1997%2E)
3. S. Bengio and Y. Bengio. Taking on the curse of dimensionality in joint distributions using neural networks. IEEE Transactions on Neural Networks, special issue on Data Mining and Knowledge Discovery, 11(3):550–557, 2000a. [find](https://scholar.google.com/scholar?q=S%2E+Bengio+and+Y%2E+Bengio%2E+Taking+on+the+curse+of+dimensionality+in+joint+distributions+using+neural+networks%2E+IEEE+Transactions+on+Neural+Networks%2C+special+issue+on+Data+Mining+and+Knowledge+Discovery%2C)
4. Y. Bengio. New distributed probabilistic language models. Technical Report 1215, Dept. IRO, Université de Montréal, 2002. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2E+New+distributed+probabilistic+language+models%2E+Technical+Report+1215%2C+Dept%2E+IRO%2C+Universit%C3%A9+de+Montr%C3%A9al%2C+2002%2E)
5. Y. Bengio and S. Bengio. Modeling high-dimensional discrete data with multi-layer neural networks. In S. A. Solla, T. K. Leen, and K-R. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages 400–406. MIT Press, 2000b. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio+and+S%2E+Bengio%2E+Modeling+high%2Ddimensional+discrete+data+with+multi%2Dlayer+neural+networks%2E+In+S%2E+A%2E+Solla%2C+T%2E+K%2E+Leen%2C+and+K%2DR%2E+M%C3%BCller%2C+editors%2C+Advances+in+Neural+Information+Processing+Syste)
6. Y. Bengio and J-S. Senécal. Quick training of probabilistic neural nets by importance sampling. In AISTATS, 2003. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio+and+J%2DS%2E+Sen%C3%A9cal%2E+Quick+training+of+probabilistic+neural+nets+by+importance+sampling%2E+In+AISTATS%2C+2003%2E)
7. A. Berger, S. Della Pietra, and V. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22:39–71, 1996. [our card](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/)
8. A. Brown and G.E. Hinton. Products of hidden markov models. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London, 2000. [find](https://scholar.google.com/scholar?q=A%2E+Brown+and+G%2EE%2E+Hinton%2E+Products+of+hidden+markov+models%2E+Technical+Report+GCNU+TR+2000%2D004%2C+Gatsby+Unit%2C+University+College+London%2C+2000%2E)
9. P.F. Brown, V.J. Della Pietra, P.V. DeSouza, J.C. Lai, and R.L. Mercer. Class-based n-gram models of natural language. Computational Linguistics, 18:467–479, 1992. [find](https://scholar.google.com/scholar?q=P%2EF%2E+Brown%2C+V%2EJ%2E+Della+Pietra%2C+P%2EV%2E+DeSouza%2C+J%2EC%2E+Lai%2C+and+R%2EL%2E+Mercer%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+Linguistics%2C+18%3A467%E2%80%93479%2C+1992%2E)
10. S.F. Chen and J.T. Goodman. An empirical study of smoothing techniques for language modeling. Computer, Speech and Language, 13(4):359–393, 1999. [find](https://scholar.google.com/scholar?q=S%2EF%2E+Chen+and+J%2ET%2E+Goodman%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2E+Computer%2C+Speech+and+Language%2C+13%284%29%3A359%E2%80%93393%2C+1999%2E)
11. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990. [find](https://scholar.google.com/scholar?q=S%2E+Deerwester%2C+S%2ET%2E+Dumais%2C+G%2EW%2E+Furnas%2C+T%2EK%2E+Landauer%2C+and+R%2E+Harshman%2E+Indexing+by+latent+semantic+analysis%2E+Journal+of+the+American+Society+for+Information+Science%2C+41%286%29%3A391%E2%80%93407%2C+1990%2E)
12. J. Dongarra, D. Walker, and The Message Passing Interface Forum. MPI: A message passing interface standard. Technical Report [http://www-unix.mcs.anl.gov/mpi](http://www-unix.mcs.anl.gov/mpi), University of Tenessee, 1995. [↗](http://www-unix.mcs.anl.gov/mpi)
13. J.L. Elman. Finding structure in time. Cognitive Science, 14:179–211, 1990. [find](https://scholar.google.com/scholar?q=J%2EL%2E+Elman%2E+Finding+structure+in+time%2E+Cognitive+Science%2C+14%3A179%E2%80%93211%2C+1990%2E)
14. C. Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, 1998. [find](https://scholar.google.com/scholar?q=C%2E+Fellbaum%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+MIT+Press%2C+1998%2E)
15. J .Goodman. A bit of progress in language modeling. Technical Report MSR-TR-2001-72, Microsoft Research, 2001. [find](https://scholar.google.com/scholar?q=J+%2EGoodman%2E+A+bit+of+progress+in+language+modeling%2E+Technical+Report+MSR%2DTR%2D2001%2D72%2C+Microsoft+Research%2C+2001%2E)
16. G.E. Hinton. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pages 1–12, Amherst 1986, 1986. Lawrence Erlbaum, Hillsdale. [find](https://scholar.google.com/scholar?q=G%2EE%2E+Hinton%2E+Learning+distributed+representations+of+concepts%2E+In+Proceedings+of+the+Eighth+Annual+Conference+of+the+Cognitive+Science+Society%2C+pages+1%E2%80%9312%2C+Amherst+1986%2C+1986%2E+Lawrence+Erlbaum%2C+Hillsd)
17. G.E. Hinton. Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London, 2000. [find](https://scholar.google.com/scholar?q=G%2EE%2E+Hinton%2E+Training+products+of+experts+by+minimizing+contrastive+divergence%2E+Technical+Report+GCNU+TR+2000%2D004%2C+Gatsby+Unit%2C+University+College+London%2C+2000%2E)
18. F. Jelinek and R. L. Mercer. Interpolated estimation of Markov source parameters from sparse data. In E. S. Gelsema and L. N. Kanal, editors, Pattern Recognition in Practice. North-Holland, Amsterdam, 1980. [find](https://scholar.google.com/scholar?q=F%2E+Jelinek+and+R%2E+L%2E+Mercer%2E+Interpolated+estimation+of+Markov+source+parameters+from+sparse+data%2E+In+E%2E+S%2E+Gelsema+and+L%2E+N%2E+Kanal%2C+editors%2C+Pattern+Recognition+in+Practice%2E+North%2DHolland%2C+Amsterdam%2C)
19. K.J. Jensen and S. Riis. Self-organizing letter code-book for text-to-phoneme neural network model. In Proceedings ICSLP, 2000. [find](https://scholar.google.com/scholar?q=K%2EJ%2E+Jensen+and+S%2E+Riis%2E+Self%2Dorganizing+letter+code%2Dbook+for+text%2Dto%2Dphoneme+neural+network+model%2E+In+Proceedings+ICSLP%2C+2000%2E)
20. S.M. Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, ASSP-35 (3):400–401, March 1987. [find](https://scholar.google.com/scholar?q=S%2EM%2E+Katz%2E+Estimation+of+probabilities+from+sparse+data+for+the+language+model+component+of+a+speech+recognizer%2E+IEEE+Transactions+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+ASSP%2D35+%283%29%3A400%E2%80%93401%2C+Mar)
21. R. Kneser and H. Ney. Improved backing-off for m-gram language modeling. In International Conference on Acoustics, Speech and Signal Processing, pages 181–184, 1995. [find](https://scholar.google.com/scholar?q=R%2E+Kneser+and+H%2E+Ney%2E+Improved+backing%2Doff+for+m%2Dgram+language+modeling%2E+In+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing%2C+pages+181%E2%80%93184%2C+1995%2E)
22. Y. LeCun, L. Bottou, G.B. Orr, and K.-R. Müller. Efficient backprop. In G.B. Orr and K.-R. Müller, editors, Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+L%2E+Bottou%2C+G%2EB%2E+Orr%2C+and+K%2E%2DR%2E+M%C3%BCller%2E+Efficient+backprop%2E+In+G%2EB%2E+Orr+and+K%2E%2DR%2E+M%C3%BCller%2C+editors%2C+Neural+Networks%3A+Tricks+of+the+Trade%2C+pages+9%E2%80%9350%2E+Springer%2C+1998%2E)
23. R. Miikkulainen and M.G. Dyer. Natural language processing with modular neural networks and distributed lexicon. Cognitive Science, 15:343–399, 1991. [find](https://scholar.google.com/scholar?q=R%2E+Miikkulainen+and+M%2EG%2E+Dyer%2E+Natural+language+processing+with+modular+neural+networks+and+distributed+lexicon%2E+Cognitive+Science%2C+15%3A343%E2%80%93399%2C+1991%2E)
24. H. Ney and R. Kneser. Improved clustering techniques for class-based statistical language modelling. In European Conference on Speech Communication and Technology (Eurospeech), pages 973–976, Berlin, 1993. [find](https://scholar.google.com/scholar?q=H%2E+Ney+and+R%2E+Kneser%2E+Improved+clustering+techniques+for+class%2Dbased+statistical+language+modelling%2E+In+European+Conference+on+Speech+Communication+and+Technology+%28Eurospeech%29%2C+pages+973%E2%80%93976%2C+Berlin%2C+)
25. T.R. Niesler, E.W.D. Whittaker, and P.C. Woodland. Comparison of part-of-speech and automatically derived category-based language models for speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pages 177–180, 1998. [find](https://scholar.google.com/scholar?q=T%2ER%2E+Niesler%2C+E%2EW%2ED%2E+Whittaker%2C+and+P%2EC%2E+Woodland%2E+Comparison+of+part%2Dof%2Dspeech+and+automatically+derived+category%2Dbased+language+models+for+speech+recognition%2E+In+International+Conference+on+Acoustic)
26. A. Paccanaro and G.E. Hinton. Extracting distributed representations of concepts and relations from positive and negative propositions. In Proceedings of the International Joint Conference on Neural Network, IJCNN’2000, Como, Italy, 2000. IEEE, New York. [find](https://scholar.google.com/scholar?q=A%2E+Paccanaro+and+G%2EE%2E+Hinton%2E+Extracting+distributed+representations+of+concepts+and+relations+from+positive+and+negative+propositions%2E+In+Proceedings+of+the+International+Joint+Conference+on+Neural+N)
27. F. Pereira, N. Tishby, and L. Lee. Distributional clustering of english words. In 30th Annual Meeting of the Association for Computational Linguistics, pages 183–190, Columbus, Ohio, 1993. [our card](https://md.research.elamur.ai/free/en/papers/pereira-1993-distributional-clustering-english-words/)
28. S. Riis and A. Krogh. Improving protein secondary structure prediction using structured neural networks and multiple sequence profiles. Journal of Computational Biology, pages 163–183, 1996. [find](https://scholar.google.com/scholar?q=S%2E+Riis+and+A%2E+Krogh%2E+Improving+protein+secondary+structure+prediction+using+structured+neural+networks+and+multiple+sequence+profiles%2E+Journal+of+Computational+Biology%2C+pages+163%E2%80%93183%2C+1996%2E)
29. J. Schmidhuber. Sequential neural text compression. IEEE Transactions on Neural Networks, 7(1): 142–146, 1996. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Sequential+neural+text+compression%2E+IEEE+Transactions+on+Neural+Networks%2C+7%281%29%3A+142%E2%80%93146%2C+1996%2E)
30. H. Schutze. Word space. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages pp. 895–902, San Mateo CA, 1993. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=H%2E+Schutze%2E+Word+space%2E+In+S%2E+J%2E+Hanson%2C+J%2E+D%2E+Cowan%2C+and+C%2E+L%2E+Giles%2C+editors%2C+Advances+in+Neural+Information+Processing+Systems+5%2C+pages+pp%2E+895%E2%80%93902%2C+San+Mateo+CA%2C+1993%2E+Morgan+Kaufmann%2E)
31. H. Schwenk and J-L. Gauvain. Connectionist language modeling for large vocabulary continuous speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pages 765–768, Orlando, Florida, 2002. [find](https://scholar.google.com/scholar?q=H%2E+Schwenk+and+J%2DL%2E+Gauvain%2E+Connectionist+language+modeling+for+large+vocabulary+continuous+speech+recognition%2E+In+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing%2C+pages+765%E2%80%93768%2C+)
32. A. Stolcke. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Statistical Language Processing, Denver, Colorado, 2002. [find](https://scholar.google.com/scholar?q=A%2E+Stolcke%2E+SRILM+%2D+an+extensible+language+modeling+toolkit%2E+In+Proceedings+of+the+International+Conference+on+Statistical+Language+Processing%2C+Denver%2C+Colorado%2C+2002%2E)
33. W. Xu and A. Rudnicky. Can artificial neural network learn language models. In International Conference on Statistical Language Processing, pages M1–13, Beijing, China, 2000. [find](https://scholar.google.com/scholar?q=W%2E+Xu+and+A%2E+Rudnicky%2E+Can+artificial+neural+network+learn+language+models%2E+In+International+Conference+on+Statistical+Language+Processing%2C+pages+M1%E2%80%9313%2C+Beijing%2C+China%2C+2000%2E)