---
title: "Deep Unsupervised Learning using Nonequilibrium Thermodynamics — Sohl-Dickstein 2015"
description: "Probabilistic models usually force a trade-off. 57 references, cited by 256 papers here."
authors: ["Jascha Sohl-Dickstein", "Eric A. Weiss", "Niru Maheswaranathan", "Surya Ganguli"]
year: 2015
venue: "arXiv:1503.03585"
source: "https://arxiv.org/abs/1503.03585"
doi: "10.48550/arxiv.1503.03585"
arxiv: "1503.03585"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Deep Unsupervised Learning using Nonequilibrium Thermodynamics

# Deep Unsupervised Learning using Nonequilibrium Thermodynamics

**Authors:** [Jascha Sohl-Dickstein](https://md.research.elamur.ai/free/en/authors/jascha-sohl-dickstein/), Eric A. Weiss, Niru Maheswaranathan and [Surya Ganguli](https://md.research.elamur.ai/free/en/authors/surya-ganguli/) · **Year:** 2015 · **Venue:** arXiv:1503.03585 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1503.03585) · [PDF](https://arxiv.org/pdf/1503.03585) · [DOI 10.48550/arxiv.1503.03585](https://doi.org/10.48550/arxiv.1503.03585) · [arXiv 1503.03585](https://arxiv.org/abs/1503.03585) · [OpenAlex](https://openalex.org/W2129069237)

**Citations:** 1,332 citations (OpenAlex, 2026-10-14) · [256](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

Probabilistic models usually force a trade-off. Simple distributions are easy to fit and evaluate but miss rich structure. Expressive ones need intractable normalization constants and expensive Monte Carlo sampling. The authors draw on non-equilibrium statistical physics and annealed importance sampling to get both properties. A fixed forward Markov chain adds small amounts of Gaussian (or binomial) noise over many steps until the data becomes a simple, tractable distribution. A generative model is then trained to undo this corruption one step at a time. When steps are small, each reversal has the same functional form as the forward kernel, so learning only requires estimating per-step means and covariances (or bit-flip rates), optimized through a lower bound on log likelihood. The resulting model, with thousands of time steps, supports exact sampling, cheap probability evaluation and straightforward posterior computation, for example in denoising and inpainting. Experiments cover a 2-D swiss roll, binary sequences, MNIST, CIFAR-10, bark textures and dead leaves images. The work laid the foundation for later diffusion-based generative models.

## Key points

- **Framework.** Generation runs a learned reverse diffusion from an identity-covariance Gaussian (or independent binomial) back to the data. The model is defined as the endpoint of this chain, so its probabilities can be evaluated analytically.
- **Dead leaves.** The model reaches state-of-the-art performance with a log-likelihood bound of 1.489 bits/pixel, compared with 1.244 bits/pixel for MCGSM (Theis et al., 2012).
- **MNIST.** Parzen-window estimates using the code from Goodfellow et al. (2014) give 317 ± 2.7 bits. This is comparable to the Deep GSN (309), adversarial net (325) and DBN (199); a "perfect model" scores 349.
- **Binary heartbeat (length-20 sequences).** Learning is nearly perfect: the bound is −2.414 bits/seq versus the true −2.322 bits/seq. On CIFAR-10, the bound improves on the Gaussian baseline by 11.5 ± 0.2 bits/pixel.
- **Implementation.** An open-source reference implementation built on Blocks and trained with RMSprop is available at github.com/Sohl-Dickstein/Diffusion-Probabilistic-Models. The paper was presented at ICML 2015.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 248 works · 214,804 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 107 works · 205,102 citations

## Cite

Cite as: Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan and Surya Ganguli. 2015. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. arXiv:1503.03585.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sohldickstein2015deep,
  title = {{Deep Unsupervised Learning using Nonequilibrium Thermodynamics}},
  author = {Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan and Surya Ganguli},
  year = {2015},
  doi = {10.48550/arxiv.1503.03585},
  eprint = {1503.03585},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1503.03585}
}
```

- [← previous SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
- [next → Effective Approaches to Attention-based Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

57 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Barron, J. T., Biggin, M. D., Arbelaez, P., Knowles, D. W., Keranen, S. V., and Malik, J. Volumetric Semantic Segmentation Using Pyramid Context Features. In 2013 IEEE International Conference on Computer Vision, pp. 3448–3455. [find](https://scholar.google.com/scholar?q=Barron%2C+J%2E+T%2E%2C+Biggin%2C+M%2E+D%2E%2C+Arbelaez%2C+P%2E%2C+Knowles%2C+D%2E+W%2E%2C+Keranen%2C+S%2E+V%2E%2C+and+Malik%2C+J%2E+Volumetric+Semantic+Segmentation+Using+Pyramid+Context+Features%2E+In+2013+IEEE+International+Conference+on+Comp)
2. IEEE, December 2013. ISBN 978-1-4799-2840-8. doi: 10.1109/ICCV.2013.428. — [doi:10.1109/iccv.2013.428](https://doi.org/10.1109/iccv.2013.428) [find](https://scholar.google.com/scholar?q=IEEE%2C+December+2013%2E+ISBN+978%2D1%2D4799%2D2840%2D8%2E+doi%3A+10%2E1109%2FICCV%2E2013%2E428%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Ficcv%2E2013%2E428%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1109%2Ficcv%2E2013%2E428%29)
3. Bengio, Y. and Thibodeau-Laufer, E. Deep generative stochastic networks trainable by backprop. arXiv preprint arXiv:1306.1091, 2013. — [arXiv:1306.1091](https://arxiv.org/abs/1306.1091) [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E+and+Thibodeau%2DLaufer%2C+E%2E+Deep+generative+stochastic+networks+trainable+by+backprop%2E+arXiv+preprint+arXiv%3A1306%2E1091%2C+2013%2E+%E2%80%94+%5BarXiv%3A1306%2E1091%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1306%2E1091%29)
4. Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. Better Mixing via Deep Representations. arXiv preprint arXiv:1207.4404, July 2012. — [arXiv:1207.4404](https://arxiv.org/abs/1207.4404) [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Mesnil%2C+G%2E%2C+Dauphin%2C+Y%2E%2C+and+Rifai%2C+S%2E+Better+Mixing+via+Deep+Representations%2E+arXiv+preprint+arXiv%3A1207%2E4404%2C+July+2012%2E+%E2%80%94+%5BarXiv%3A1207%2E4404%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1207%2E4404%29)
5. Bergstra, J. and Breuleux, O. Theano: a CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy), 2010. [find](https://scholar.google.com/scholar?q=Bergstra%2C+J%2E+and+Breuleux%2C+O%2E+Theano%3A+a+CPU+and+GPU+math+expression+compiler%2E+Proceedings+of+the+Python+for+Scientific+Computing+Conference+%28SciPy%29%2C+2010%2E)
6. Besag, J. Statistical Analysis of Non-Lattice Data. The Statistician, 24(3), 179-195, 1975. [find](https://scholar.google.com/scholar?q=Besag%2C+J%2E+Statistical+Analysis+of+Non%2DLattice+Data%2E+The+Statistician%2C+24%283%29%2C+179%2D195%2C+1975%2E)
7. Bishop, C., Svensén, M., and Williams, C. GTM: The generative topographic mapping. Neural computation, 1998. [find](https://scholar.google.com/scholar?q=Bishop%2C+C%2E%2C+Svense%CC%81n%2C+M%2E%2C+and+Williams%2C+C%2E+GTM%3A+The+generative+topographic+mapping%2E+Neural+computation%2C+1998%2E)
8. Bornschein, J. and Bengio, Y. Reweighted Wake-Sleep. International Conference on Learning Representations, June 2015. [find](https://scholar.google.com/scholar?q=Bornschein%2C+J%2E+and+Bengio%2C+Y%2E+Reweighted+Wake%2DSleep%2E+International+Conference+on+Learning+Representations%2C+June+2015%2E)
9. Burda, Y., Grosse, R. B., and Salakhutdinov, R. Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing. arXiv:1412.8566, December 2014. — [arXiv:1412.8566](https://arxiv.org/abs/1412.8566) [find](https://scholar.google.com/scholar?q=Burda%2C+Y%2E%2C+Grosse%2C+R%2E+B%2E%2C+and+Salakhutdinov%2C+R%2E+Accurate+and+Conservative+Estimates+of+MRF+Log%2Dlikelihood+using+Reverse+Annealing%2E+arXiv%3A1412%2E8566%2C+December+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E8566%5D%28https%3A%2F%2Farxiv%2Eorg)
10. Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. The helmholtz machine. Neural computation, 7(5):889–904, 1995. [find](https://scholar.google.com/scholar?q=Dayan%2C+P%2E%2C+Hinton%2C+G%2E+E%2E%2C+Neal%2C+R%2E+M%2E%2C+and+Zemel%2C+R%2E+S%2E+The+helmholtz+machine%2E+Neural+computation%2C+7%285%29%3A889%E2%80%93904%2C+1995%2E)
11. Dinh, L., Krueger, D., and Bengio, Y. NICE: Non-linear Independent Components Estimation. arXiv:1410.8516, pp. 11, October 2014. — [arXiv:1410.8516](https://arxiv.org/abs/1410.8516) [find](https://scholar.google.com/scholar?q=Dinh%2C+L%2E%2C+Krueger%2C+D%2E%2C+and+Bengio%2C+Y%2E+NICE%3A+Non%2Dlinear+Independent+Components+Estimation%2E+arXiv%3A1410%2E8516%2C+pp%2E+11%2C+October+2014%2E+%E2%80%94+%5BarXiv%3A1410%2E8516%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1410%2E8516%29)
12. Feller, W. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the [First] Berkeley Symposium on Mathematical Statistics and Probability. The Regents of the University of California, 1949. [find](https://scholar.google.com/scholar?q=Feller%2C+W%2E+On+the+theory+of+stochastic+processes%2C+with+particular+reference+to+applications%2E+In+Proceedings+of+the+%5BFirst%5D+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2E+The+Regents+of)
13. Gershman, S. J. and Blei, D. M. A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56(1):1–12, 2012. [find](https://scholar.google.com/scholar?q=Gershman%2C+S%2E+J%2E+and+Blei%2C+D%2E+M%2E+A+tutorial+on+Bayesian+nonparametric+models%2E+Journal+of+Mathematical+Psychology%2C+56%281%29%3A1%E2%80%9312%2C+2012%2E)
14. Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. [find](https://scholar.google.com/scholar?q=Gneiting%2C+T%2E+and+Raftery%2C+A%2E+E%2E+Strictly+proper+scoring+rules%2C+prediction%2C+and+estimation%2E+Journal+of+the+American+Statistical+Association%2C+102%28477%29%3A359%E2%80%93378%2C+2007%2E)
15. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative Adversarial Nets. Advances in Neural Information Processing Systems, 2014. [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2E+J%2E%2C+Pouget%2DAbadie%2C+J%2E%2C+Mirza%2C+M%2E%2C+Xu%2C+B%2E%2C+Warde%2DFarley%2C+D%2E%2C+Ozair%2C+S%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+Generative+Adversarial+Nets%2E+Advances+in+Neural+Information+Processing+Systems%2C+2014)
16. Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep AutoRegressive Networks. arXiv preprint arXiv:1310.8499, October 2013. — [arXiv:1310.8499](https://arxiv.org/abs/1310.8499) [find](https://scholar.google.com/scholar?q=Gregor%2C+K%2E%2C+Danihelka%2C+I%2E%2C+Mnih%2C+A%2E%2C+Blundell%2C+C%2E%2C+and+Wierstra%2C+D%2E+Deep+AutoRegressive+Networks%2E+arXiv+preprint+arXiv%3A1310%2E8499%2C+October+2013%2E+%E2%80%94+%5BarXiv%3A1310%2E8499%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1310%2E8499%29)
17. Grosse, R. B., Maddison, C. J., and Salakhutdinov, R. Annealing between distributions by averaging moments. In Advances in Neural Information Processing Systems, pp. 2769–2777, 2013. [find](https://scholar.google.com/scholar?q=Grosse%2C+R%2E+B%2E%2C+Maddison%2C+C%2E+J%2E%2C+and+Salakhutdinov%2C+R%2E+Annealing+between+distributions+by+averaging+moments%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2769%E2%80%932777%2C+2013%2E)
18. Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8): 1771–1800, 2002. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E+E%2E+Training+products+of+experts+by+minimizing+contrastive+divergence%2E+Neural+Computation%2C+14%288%29%3A+1771%E2%80%931800%2C+2002%2E)
19. Hinton, G. E. The wake-sleep algorithm for unsupervised neural networks ). Science, 1995. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E+E%2E+The+wake%2Dsleep+algorithm+for+unsupervised+neural+networks+%29%2E+Science%2C+1995%2E)
20. Hyvärinen, A. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6:695–709, 2005. [find](https://scholar.google.com/scholar?q=Hyva%CC%88rinen%2C+A%2E+Estimation+of+non%2Dnormalized+statistical+models+using+score+matching%2E+Journal+of+Machine+Learning+Research%2C+6%3A695%E2%80%93709%2C+2005%2E)
21. Jarzynski, C. Equilibrium free-energy differences from nonequilibrium measurements: A master-equation approach. Physical Review E, January 1997. [find](https://scholar.google.com/scholar?q=Jarzynski%2C+C%2E+Equilibrium+free%2Denergy+differences+from+nonequilibrium+measurements%3A+A+master%2Dequation+approach%2E+Physical+Review+E%2C+January+1997%2E)
22. Jarzynski, C. Equalities and inequalities: irreversibility and the second law of thermodynamics at the nanoscale. Annu. Rev. Condens. Matter Phys., 2011. [find](https://scholar.google.com/scholar?q=Jarzynski%2C+C%2E+Equalities+and+inequalities%3A+irreversibility+and+the+second+law+of+thermodynamics+at+the+nanoscale%2E+Annu%2E+Rev%2E+Condens%2E+Matter+Phys%2E%2C+2011%2E)
23. Jeulin, D. Dead leaves models: from space tesselation to random functions. Proc. of the Symposium on the Advances in the Theory and Applications of Random Sets, 1997. [find](https://scholar.google.com/scholar?q=Jeulin%2C+D%2E+Dead+leaves+models%3A+from+space+tesselation+to+random+functions%2E+Proc%2E+of+the+Symposium+on+the+Advances+in+the+Theory+and+Applications+of+Random+Sets%2C+1997%2E)
24. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999. [find](https://scholar.google.com/scholar?q=Jordan%2C+M%2E+I%2E%2C+Ghahramani%2C+Z%2E%2C+Jaakkola%2C+T%2E+S%2E%2C+and+Saul%2C+L%2E+K%2E+An+introduction+to+variational+methods+for+graphical+models%2E+Machine+learning%2C+37%282%29%3A183%E2%80%93233%2C+1999%2E)
25. Kavukcuoglu, K., Ranzato, M., and LeCun, Y. Fast inference in sparse coding algorithms with applications to object recognition. arXiv preprint arXiv:1010.3467, 2010. — [arXiv:1010.3467](https://arxiv.org/abs/1010.3467) [find](https://scholar.google.com/scholar?q=Kavukcuoglu%2C+K%2E%2C+Ranzato%2C+M%2E%2C+and+LeCun%2C+Y%2E+Fast+inference+in+sparse+coding+algorithms+with+applications+to+object+recognition%2E+arXiv+preprint+arXiv%3A1010%2E3467%2C+2010%2E+%E2%80%94+%5BarXiv%3A1010%2E3467%5D%28https%3A%2F%2Farxiv%2E)
26. Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. International Conference on Learning Representations, December 2013. [our card](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/)
27. Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Computer Science Department University of Toronto Tech. Rep., 2009. [find](https://scholar.google.com/scholar?q=Krizhevsky%2C+A%2E+and+Hinton%2C+G%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Computer+Science+Department+University+of+Toronto+Tech%2E+Rep%2E%2C+2009%2E)
28. Langevin, P. Sur la théorie du mouvement brownien. CR Acad. Sci. Paris, 146(530-533), 1908. [find](https://scholar.google.com/scholar?q=Langevin%2C+P%2E+Sur+la+the%CC%81orie+du+mouvement+brownien%2E+CR+Acad%2E+Sci%2E+Paris%2C+146%28530%2D533%29%2C+1908%2E)
29. Larochelle, H. and Murray, I. The neural autoregressive distribution estimator. Journal of Machine Learning Research, 2011. [find](https://scholar.google.com/scholar?q=Larochelle%2C+H%2E+and+Murray%2C+I%2E+The+neural+autoregressive+distribution+estimator%2E+Journal+of+Machine+Learning+Research%2C+2011%2E)
30. Lazebnik, S., Schmid, C., and Ponce, J. A sparse texture representation using local affine regions. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27 (8):1265–1278, 2005. [find](https://scholar.google.com/scholar?q=Lazebnik%2C+S%2E%2C+Schmid%2C+C%2E%2C+and+Ponce%2C+J%2E+A+sparse+texture+representation+using+local+affine+regions%2E+Pattern+Analysis+and+Machine+Intelligence%2C+IEEE+Transactions+on%2C+27+%288%29%3A1265%E2%80%931278%2C+2005%2E)
31. LeCun, Y. and Cortes, C. The MNIST database of handwritten digits. 1998. [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E+and+Cortes%2C+C%2E+The+MNIST+database+of+handwritten+digits%2E+1998%2E)
32. Lee, A., Mumford, D., and Huang, J. Occlusion models for natural images: A statistical study of a scale-invariant dead leaves model. International Journal of Computer Vision, 2001. [find](https://scholar.google.com/scholar?q=Lee%2C+A%2E%2C+Mumford%2C+D%2E%2C+and+Huang%2C+J%2E+Occlusion+models+for+natural+images%3A+A+statistical+study+of+a+scale%2Dinvariant+dead+leaves+model%2E+International+Journal+of+Computer+Vision%2C+2001%2E)
33. Lyu, S. Unifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction. Advances in Neural Information Processing Systems 24, pp. 64–72, 2011. [find](https://scholar.google.com/scholar?q=Lyu%2C+S%2E+Unifying+Non%2DMaximum+Likelihood+Learning+Objectives+with+Minimum+KL+Contraction%2E+Advances+in+Neural+Information+Processing+Systems+24%2C+pp%2E+64%E2%80%9372%2C+2011%2E)
34. MacKay, D. Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 1995. [find](https://scholar.google.com/scholar?q=MacKay%2C+D%2E+Bayesian+neural+networks+and+density+networks%2E+Nuclear+Instruments+and+Methods+in+Physics+Research+Section+A%3A+Accelerators%2C+Spectrometers%2C+Detectors+and+Associated+Equipment%2C+1995%2E)
35. Murphy, K. P., Weiss, Y., and Jordan, M. I. Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pp. 467–475. Morgan Kaufmann Publishers Inc., 1999. [find](https://scholar.google.com/scholar?q=Murphy%2C+K%2E+P%2E%2C+Weiss%2C+Y%2E%2C+and+Jordan%2C+M%2E+I%2E+Loopy+belief+propagation+for+approximate+inference%3A+An+empirical+study%2E+In+Proceedings+of+the+Fifteenth+conference+on+Uncertainty+in+artificial+intelligence)
36. Neal, R. Annealed importance sampling. Statistics and Computing, January 2001. [find](https://scholar.google.com/scholar?q=Neal%2C+R%2E+Annealed+importance+sampling%2E+Statistics+and+Computing%2C+January+2001%2E)
37. Ozair, S. and Bengio, Y. Deep Directed Generative Autoencoders. arXiv:1410.0630, October 2014. — [arXiv:1410.0630](https://arxiv.org/abs/1410.0630) [find](https://scholar.google.com/scholar?q=Ozair%2C+S%2E+and+Bengio%2C+Y%2E+Deep+Directed+Generative+Autoencoders%2E+arXiv%3A1410%2E0630%2C+October+2014%2E+%E2%80%94+%5BarXiv%3A1410%2E0630%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1410%2E0630%29)
38. Parry, M., Dawid, A. P., Lauritzen, S., and Others. Proper local scoring rules. The Annals of Statistics, 40(1):561– 592, 2012. [find](https://scholar.google.com/scholar?q=Parry%2C+M%2E%2C+Dawid%2C+A%2E+P%2E%2C+Lauritzen%2C+S%2E%2C+and+Others%2E+Proper+local+scoring+rules%2E+The+Annals+of+Statistics%2C+40%281%29%3A561%E2%80%93+592%2C+2012%2E)
39. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 31st International Conference on Machine Learning (ICML-14), January 2014. [our card](https://md.research.elamur.ai/free/en/papers/rezende-2014-stochastic-backpropagation-approximate-inference-deep/)
40. Rippel, O. and Adams, R. P. High-Dimensional Probability Estimation with Deep Density Models. arXiv:1410.8516, pp. 12, February 2013. — [arXiv:1410.8516](https://arxiv.org/abs/1410.8516) [find](https://scholar.google.com/scholar?q=Rippel%2C+O%2E+and+Adams%2C+R%2E+P%2E+High%2DDimensional+Probability+Estimation+with+Deep+Density+Models%2E+arXiv%3A1410%2E8516%2C+pp%2E+12%2C+February+2013%2E+%E2%80%94+%5BarXiv%3A1410%2E8516%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1410%2E8516%29)
41. Schmidhuber, J. Learning factorial codes by predictability minimization. Neural Computation, 1992. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E+Learning+factorial+codes+by+predictability+minimization%2E+Neural+Computation%2C+1992%2E)
42. Sminchisescu, C., Kanaujia, A., and Metaxas, D. Learning joint top-down and bottom-up processes for 3D visual inference. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 2, pp. 1743–1752. [find](https://scholar.google.com/scholar?q=Sminchisescu%2C+C%2E%2C+Kanaujia%2C+A%2E%2C+and+Metaxas%2C+D%2E+Learning+joint+top%2Ddown+and+bottom%2Dup+processes+for+3D+visual+inference%2E+In+Computer+Vision+and+Pattern+Recognition%2C+2006+IEEE+Computer+Society+Conferen)
43. Sohl-Dickstein, J., Battaglino, P., and DeWeese, M. New Method for Parameter Estimation in Probabilistic Models: Minimum Probability Flow. Physical Review Letters, 107(22):11–14, November 2011a. ISSN 0031-9007. doi: 10.1103/PhysRevLett.107.220601. — [doi:10.1103/physrevlett.107.220601](https://doi.org/10.1103/physrevlett.107.220601) [find](https://scholar.google.com/scholar?q=Sohl%2DDickstein%2C+J%2E%2C+Battaglino%2C+P%2E%2C+and+DeWeese%2C+M%2E+New+Method+for+Parameter+Estimation+in+Probabilistic+Models%3A+Minimum+Probability+Flow%2E+Physical+Review+Letters%2C+107%2822%29%3A11%E2%80%9314%2C+November+2011a%2E+ISSN+)
44. Sohl-Dickstein, J., Battaglino, P. B., and DeWeese, M. R. Minimum Probability Flow Learning. International Conference on Machine Learning, 107(22):11– 14, November 2011b. ISSN 0031-9007. doi: 10.1103/PhysRevLett.107.220601. — [doi:10.1103/physrevlett.107.220601](https://doi.org/10.1103/physrevlett.107.220601) [find](https://scholar.google.com/scholar?q=Sohl%2DDickstein%2C+J%2E%2C+Battaglino%2C+P%2E+B%2E%2C+and+DeWeese%2C+M%2E+R%2E+Minimum+Probability+Flow+Learning%2E+International+Conference+on+Machine+Learning%2C+107%2822%29%3A11%E2%80%93+14%2C+November+2011b%2E+ISSN+0031%2D9007%2E+doi%3A+10%2E1103%2F)
45. Sohl-Dickstein, J., Poole, B., and Ganguli, S. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 604–612, 2014. [find](https://scholar.google.com/scholar?q=Sohl%2DDickstein%2C+J%2E%2C+Poole%2C+B%2E%2C+and+Ganguli%2C+S%2E+Fast+large%2Dscale+optimization+by+unifying+stochastic+gradient+and+quasi%2DNewton+methods%2E+In+Proceedings+of+the+31st+International+Conference+on+Machine+Le)
46. Spinney, R. and Ford, I. Fluctuation Relations : A Pedagogical Overview. arXiv preprint arXiv:1201.6381, pp. 3–56, 2013. — [arXiv:1201.6381](https://arxiv.org/abs/1201.6381) [find](https://scholar.google.com/scholar?q=Spinney%2C+R%2E+and+Ford%2C+I%2E+Fluctuation+Relations+%3A+A+Pedagogical+Overview%2E+arXiv+preprint+arXiv%3A1201%2E6381%2C+pp%2E+3%E2%80%9356%2C+2013%2E+%E2%80%94+%5BarXiv%3A1201%2E6381%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1201%2E6381%29)
47. Stuhlmüller, A., Taylor, J., and Goodman, N. Learning stochastic inverses. Advances in Neural Information Processing Systems, 2013. [find](https://scholar.google.com/scholar?q=Stuhlmu%CC%88ller%2C+A%2E%2C+Taylor%2C+J%2E%2C+and+Goodman%2C+N%2E+Learning+stochastic+inverses%2E+Advances+in+Neural+Information+Processing+Systems%2C+2013%2E)
48. Suykens, J. and Vandewalle, J. Nonconvex optimization using a Fokker-Planck learning machine. In 12th European Conference on Circuit Theory and Design, 1995. [find](https://scholar.google.com/scholar?q=Suykens%2C+J%2E+and+Vandewalle%2C+J%2E+Nonconvex+optimization+using+a+Fokker%2DPlanck+learning+machine%2E+In+12th+European+Conference+on+Circuit+Theory+and+Design%2C+1995%2E)
49. T, P. Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model. J. Phys. A: Math. Gen. 15 1971, 1982. [find](https://scholar.google.com/scholar?q=T%2C+P%2E+Convergence+condition+of+the+TAP+equation+for+the+infinite%2Dranged+Ising+spin+glass+model%2E+J%2E+Phys%2E+A%3A+Math%2E+Gen%2E+15+1971%2C+1982%2E)
50. Tanaka, T. Mean-field theory of Boltzmann machine learning. Physical Review Letters E, January 1998. [find](https://scholar.google.com/scholar?q=Tanaka%2C+T%2E+Mean%2Dfield+theory+of+Boltzmann+machine+learning%2E+Physical+Review+Letters+E%2C+January+1998%2E)
51. Theis, L., Hosseini, R., and Bethge, M. Mixtures of conditional Gaussian scale mixtures applied to multiscale image representations. PloS one, 7(7):e39857, 2012. [find](https://scholar.google.com/scholar?q=Theis%2C+L%2E%2C+Hosseini%2C+R%2E%2C+and+Bethge%2C+M%2E+Mixtures+of+conditional+Gaussian+scale+mixtures+applied+to+multiscale+image+representations%2E+PloS+one%2C+7%287%29%3Ae39857%2C+2012%2E)
52. Theis, L., van den Oord, A., and Bethge, M. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844, 2015. — [arXiv:1511.01844](https://arxiv.org/abs/1511.01844) [find](https://scholar.google.com/scholar?q=Theis%2C+L%2E%2C+van+den+Oord%2C+A%2E%2C+and+Bethge%2C+M%2E+A+note+on+the+evaluation+of+generative+models%2E+arXiv+preprint+arXiv%3A1511%2E01844%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E01844%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E01844%29)
53. Uria, B., Murray, I., and Larochelle, H. RNADE: The real-valued neural autoregressive density-estimator. Advances in Neural Information Processing Systems, 2013a. [find](https://scholar.google.com/scholar?q=Uria%2C+B%2E%2C+Murray%2C+I%2E%2C+and+Larochelle%2C+H%2E+RNADE%3A+The+real%2Dvalued+neural+autoregressive+density%2Destimator%2E+Advances+in+Neural+Information+Processing+Systems%2C+2013a%2E)
54. Uria, B., Murray, I., and Larochelle, H. A Deep and Tractable Density Estimator. arXiv:1310.1757, pp. 9, October 2013b. van Merriënboer, B., Chorowski, J., Serdyuk, D., Bengio, Y., Bogdanov, D., Dumoulin, V., and Warde-Farley, D. Blocks and Fuel. — [arXiv:1310.1757](https://arxiv.org/abs/1310.1757) [find](https://scholar.google.com/scholar?q=Uria%2C+B%2E%2C+Murray%2C+I%2E%2C+and+Larochelle%2C+H%2E+A+Deep+and+Tractable+Density+Estimator%2E+arXiv%3A1310%2E1757%2C+pp%2E+9%2C+October+2013b%2E+van+Merrie%CC%88nboer%2C+B%2E%2C+Chorowski%2C+J%2E%2C+Serdyuk%2C+D%2E%2C+Bengio%2C+Y%2E%2C+Bogdanov%2C+D%2E%2C+Dumo)
55. Zenodo, May 2015. doi: 10.5281/zenodo.17721. — [doi:10.5281/zenodo.17721](https://doi.org/10.5281/zenodo.17721) [find](https://scholar.google.com/scholar?q=Zenodo%2C+May+2015%2E+doi%3A+10%2E5281%2Fzenodo%2E17721%2E+%E2%80%94+%5Bdoi%3A10%2E5281%2Fzenodo%2E17721%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E5281%2Fzenodo%2E17721%29)
56. Welling, M. and Hinton, G. A new learning algorithm for mean field Boltzmann machines. Lecture Notes in Computer Science, January 2002. [find](https://scholar.google.com/scholar?q=Welling%2C+M%2E+and+Hinton%2C+G%2E+A+new+learning+algorithm+for+mean+field+Boltzmann+machines%2E+Lecture+Notes+in+Computer+Science%2C+January+2002%2E)
57. Yao, L., Ozair, S., Cho, K., and Bengio, Y. On the Equivalence Between Deep NADE and Generative Stochastic Networks. In Machine Learning and Knowledge Discovery in Databases, pp. 322–336. Springer, 2014. [find](https://scholar.google.com/scholar?q=Yao%2C+L%2E%2C+Ozair%2C+S%2E%2C+Cho%2C+K%2E%2C+and+Bengio%2C+Y%2E+On+the+Equivalence+Between+Deep+NADE+and+Generative+Stochastic+Networks%2E+In+Machine+Learning+and+Knowledge+Discovery+in+Databases%2C+pp%2E+322%E2%80%93336%2E+Springer%2C+20)