---
title: "Incentivizing Exploration In Reinforcement Learning With Deep"
description: "The authors replace visitation counts with the \"surprise\" of a learned dynamics model. 25 references, cited by 33 papers here."
authors: ["Bradly C. Stadie", "Sergey Levine", "Pieter Abbeel"]
year: 2015
venue: "arXiv:1507.00814"
source: "https://arxiv.org/abs/1507.00814"
doi: "10.48550/arxiv.1507.00814"
arxiv: "1507.00814"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stadie-2015-incentivizing-exploration-reinforcement-learning-deep/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stadie-2015-incentivizing-exploration-reinforcement-learning-deep/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

# Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

**Authors:** Bradly C. Stadie, [Sergey Levine](https://md.research.elamur.ai/free/en/authors/sergey-levine/) and [Pieter Abbeel](https://md.research.elamur.ai/free/en/authors/pieter-abbeel/) · **Year:** 2015 · **Venue:** arXiv:1507.00814 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1507.00814) · [PDF](https://arxiv.org/pdf/1507.00814) · [DOI 10.48550/arxiv.1507.00814](https://doi.org/10.48550/arxiv.1507.00814) · [arXiv 1507.00814](https://arxiv.org/abs/1507.00814) · [OpenAlex](https://openalex.org/W779494576)

**Citations:** 280 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/stadie-2015-incentivizing-exploration-reinforcement-learning-deep/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Exploration in reinforcement learning with high-dimensional observations is still hard. Count-based bonus methods such as MBIE-EB and BEB come with theoretical guarantees, but they need to enumerate states. Large-scale agents therefore usually fall back on epsilon-greedy. The authors replace visitation counts with the "surprise" of a learned dynamics model.

Atari frames are first compressed by an autoencoder. A neural network then predicts the next encoded frame from the current encoding and the chosen action. The squared prediction error is normalized by the largest error seen so far, decayed over time, and added to the game reward as a novelty bonus. The scheme works with any online RL learner. Here it is combined with DQN on Arcade Learning Environment games and compared with Boltzman exploration and Thompson sampling.

Model-based bonuses gave the most consistent improvements, especially on games where humans far outperform earlier agents, and they also accelerated learning. To measure learning speed, the paper introduces AUC-100, a normalized area under the score curve over the first 100 epochs.

## Key points
- **Novelty signal:** the novelty of a transition is the normalized squared error of the dynamics predictor in the autoencoded space. Predicting raw pixels produced nearly uniform, uninformative bonuses.
- **Encoder design:** an eight-layer autoencoder is used, taking its sixth layer as the state code. On Seaquest, this layer beat the fourth-layer bottleneck.
- **Training schedule:** the policy updates every step, while the dynamics model is refit once per epoch (50,000 observations). The encoder is retrained every 5 epochs.
- **AUC-100 results:** examples include Bowling (DQN 0.964, static-AE bonus 1.493, dynamic-AE bonus 1.836), Frostbite (0.573 vs 0.971 with static AE), and Seaquest (0.16 vs 0.265 with dynamic AE). Every method scored 0 on Montezuma.
- **Model learning:** across all runs, the mean normalized prediction residual after 100 epochs fell to roughly half of its peak. This suggests the models genuinely learned game dynamics.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 145 works

## Cite

Cite as: Bradly C. Stadie, Sergey Levine and Pieter Abbeel. 2015. Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models. arXiv:1507.00814.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{stadie2015incentivizing,
  title = {{Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models}},
  author = {Bradly C. Stadie, Sergey Levine and Pieter Abbeel},
  year = {2015},
  doi = {10.48550/arxiv.1507.00814},
  eprint = {1507.00814},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1507.00814}
}
```

- [← previous Multi-task Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
- [next → Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. L. Strehl and M. L. Littman, An Analysis of Model-Based Interval Estimation for Markov Decision Processes. Journal of Computer and System Sciences, 74, 12091331. [find](https://scholar.google.com/scholar?q=A%2E+L%2E+Strehl+and+M%2E+L%2E+Littman%2C+An+Analysis+of+Model%2DBased+Interval+Estimation+for+Markov+Decision+Processes%2E+Journal+of+Computer+and+System+Sciences%2C+74%2C+12091331%2E)
2. J. Z. Kolter and A. Y. Ng, Near-Bayesian Exploration in Polynomial Time Proceedings of the 26th Annual International Conference on Machine Learning, pp. 18, 2009. [find](https://scholar.google.com/scholar?q=J%2E+Z%2E+Kolter+and+A%2E+Y%2E+Ng%2C+Near%2DBayesian+Exploration+in+Polynomial+Time+Proceedings+of+the+26th+Annual+International+Conference+on+Machine+Learning%2C+pp%2E+18%2C+2009%2E)
3. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, et al. Human-level Control Through Deep Reinforcement Learning. Nature, 518(7540):529533, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+A%2E+Rusu%2C+et+al%2E+Human%2Dlevel+Control+Through+Deep+Reinforcement+Learning%2E+Nature%2C+518%287540%29%3A529533%2C+2015%2E)
4. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer series in statistics. Springer, New York, 2001. [find](https://scholar.google.com/scholar?q=T%2E+Hastie%2C+R%2E+Tibshirani%2C+and+J%2E+Friedman%2C+The+Elements+of+Statistical+Learning%2E+Springer+series+in+statistics%2E+Springer%2C+New+York%2C+2001%2E)
5. M. Belkin, P. Niyogi and V. Sindhwani, Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples. JMLR, vol. 7, pp. 2399-2434, Nov. 2006. [find](https://scholar.google.com/scholar?q=M%2E+Belkin%2C+P%2E+Niyogi+and+V%2E+Sindhwani%2C+Manifold+Regularization%3A+A+Geometric+Framework+for+Learning+from+Labeled+and+Unlabeled+Examples%2E+JMLR%2C+vol%2E+7%2C+pp%2E+2399%2D2434%2C+Nov%2E+2006%2E)
6. G. E. Hinton and R. Salakhutdinov, Reducing the dimensionality of data with neural networks. Science, 313, 504–507, 2006. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Hinton+and+R%2E+Salakhutdinov%2C+Reducing+the+dimensionality+of+data+with+neural+networks%2E+Science%2C+313%2C+504%E2%80%93507%2C+2006%2E)
7. R. I. Brafman and M. Tennenholtz, R-max, a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 2002. [find](https://scholar.google.com/scholar?q=R%2E+I%2E+Brafman+and+M%2E+Tennenholtz%2C+R%2Dmax%2C+a+general+polynomial+time+algorithm+for+near%2Doptimal+reinforcement+learning%2E+Journal+of+Machine+Learning+Research%2C+2002%2E)
8. M. Kearns and D. Koller, Efficient reinforcement learning in factored MDPs. Proc. IJCAI, 1999. [find](https://scholar.google.com/scholar?q=M%2E+Kearns+and+D%2E+Koller%2C+Efficient+reinforcement+learning+in+factored+MDPs%2E+Proc%2E+IJCAI%2C+1999%2E)
9. M. Kearns and S. Singh, Near-optimal reinforcement learning in polynomial time. Machine Learning Journal, 2002. [find](https://scholar.google.com/scholar?q=M%2E+Kearns+and+S%2E+Singh%2C+Near%2Doptimal+reinforcement+learning+in+polynomial+time%2E+Machine+Learning+Journal%2C+2002%2E)
10. W. D. Smart and L. P. Kaelbling, Practical reinforcement learning in continuous spaces. Proc. ICML, 2000. [find](https://scholar.google.com/scholar?q=W%2E+D%2E+Smart+and+L%2E+P%2E+Kaelbling%2C+Practical+reinforcement+learning+in+continuous+spaces%2E+Proc%2E+ICML%2C+2000%2E)
11. J. Sorg, S. Singh, R. L. Lewis, Variance-Based Rewards for Approximate Bayesian Reinforcement Learning. Proc. UAI, 2010. [find](https://scholar.google.com/scholar?q=J%2E+Sorg%2C+S%2E+Singh%2C+R%2E+L%2E+Lewis%2C+Variance%2DBased+Rewards+for+Approximate+Bayesian+Reinforcement+Learning%2E+Proc%2E+UAI%2C+2010%2E)
12. M. Lopes, T. Lang, M. Toussaint and P.-Y. Oudeyer, Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress. NIPS, 2012. [find](https://scholar.google.com/scholar?q=M%2E+Lopes%2C+T%2E+Lang%2C+M%2E+Toussaint+and+P%2E%2DY%2E+Oudeyer%2C+Exploration+in+Model%2Dbased+Reinforcement+Learning+by+Empirically+Estimating+Learning+Progress%2E+NIPS%2C+2012%2E)
13. M. Geist and O. Pietquin, Managing Uncertainty within Value Function Approximation in Reinforcement Learning. W. on Active Learning and Experimental Design, 2010. [find](https://scholar.google.com/scholar?q=M%2E+Geist+and+O%2E+Pietquin%2C+Managing+Uncertainty+within+Value+Function+Approximation+in+Reinforcement+Learning%2E+W%2E+on+Active+Learning+and+Experimental+Design%2C+2010%2E)
14. M. Araya, O. Buffet, and V. Thomas. Near-optimal BRL Using Optimistic Local Transitions. (ICML-12), ser. ICML 12, J. Langford and J. Pineau, Eds. New York, NY, USA: Omnipress, Jul. 2012, pp. 97-104. [find](https://scholar.google.com/scholar?q=M%2E+Araya%2C+O%2E+Buffet%2C+and+V%2E+Thomas%2E+Near%2Doptimal+BRL+Using+Optimistic+Local+Transitions%2E+%28ICML%2D12%29%2C+ser%2E+ICML+12%2C+J%2E+Langford+and+J%2E+Pineau%2C+Eds%2E+New+York%2C+NY%2C+USA%3A+Omnipress%2C+Jul%2E+2012%2C+pp%2E+97%2D104%2E)
15. F. Doshi-Velez, D. Wingate, N. Roy, and J. Tenenbaum, Nonparametric Bayesian Policy Priors for Reinforcement Learning. NIPS, 2014. [find](https://scholar.google.com/scholar?q=F%2E+Doshi%2DVelez%2C+D%2E+Wingate%2C+N%2E+Roy%2C+and+J%2E+Tenenbaum%2C+Nonparametric+Bayesian+Policy+Priors+for+Reinforcement+Learning%2E+NIPS%2C+2014%2E)
16. T. Lang, M. Toussaint, K. Keristing, Exploration in relational domains for model-based reinforcement learning Proc. AAMAS, 2014. [find](https://scholar.google.com/scholar?q=T%2E+Lang%2C+M%2E+Toussaint%2C+K%2E+Keristing%2C+Exploration+in+relational+domains+for+model%2Dbased+reinforcement+learning+Proc%2E+AAMAS%2C+2014%2E)
17. Juergen Schmidhuber Developmental Robotics, Optimal Artificial Curiosity, Creativity, Music, and the Fine Arts. . Connection Science, vol. 18 (2), p 173-187. 2006. [find](https://scholar.google.com/scholar?q=Juergen+Schmidhuber+Developmental+Robotics%2C+Optimal+Artificial+Curiosity%2C+Creativity%2C+Music%2C+and+the+Fine+Arts%2E+%2E+Connection+Science%2C+vol%2E+18+%282%29%2C+p+173%2D187%2E+2006%2E)
18. J. Pazis and R. Parr, PAC Optimal Exploration in Continuous Space Markov Decision Processes. Proc. AAAI, 2013. [find](https://scholar.google.com/scholar?q=J%2E+Pazis+and+R%2E+Parr%2C+PAC+Optimal+Exploration+in+Continuous+Space+Markov+Decision+Processes%2E+Proc%2E+AAAI%2C+2013%2E)
19. Kakade, S., Kearns, M., and Langford, J. (2003). Exploration in metric state spaces. Proc. ICML. [find](https://scholar.google.com/scholar?q=Kakade%2C+S%2E%2C+Kearns%2C+M%2E%2C+and+Langford%2C+J%2E+%282003%29%2E+Exploration+in+metric+state+spaces%2E+Proc%2E+ICML%2E)
20. A. Guez, D. Silver, P. Dayan, Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search. NIPS, 2014. [find](https://scholar.google.com/scholar?q=A%2E+Guez%2C+D%2E+Silver%2C+P%2E+Dayan%2C+Efficient+Bayes%2DAdaptive+Reinforcement+Learning+using+Sample%2DBased+Search%2E+NIPS%2C+2014%2E)
21. X. Guo, S. Singh, H. Lee, R. Lewis, X. Wang, Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning. NIPS, 2014. [find](https://scholar.google.com/scholar?q=X%2E+Guo%2C+S%2E+Singh%2C+H%2E+Lee%2C+R%2E+Lewis%2C+X%2E+Wang%2C+Deep+Learning+for+Real%2DTime+Atari+Game+Play+Using+Offline+Monte%2DCarlo+Tree+Search+Planning%2E+NIPS%2C+2014%2E)
22. Y Gal, Z Ghahramani. Dropout as a Bayesian approximation: Insights and applications Deep Learning Workshop, ICML [find](https://scholar.google.com/scholar?q=Y+Gal%2C+Z+Ghahramani%2E+Dropout+as+a+Bayesian+approximation%3A+Insights+and+applications+Deep+Learning+Workshop%2C+ICML)
23. David Carmel and Shaul Markovitch. Exploration Strategies for Model-based Learning in Multi-agent Systems Autonomous Agents and Multi-Agent Systems Volume 2, Issue 2 , pp 141-172 [find](https://scholar.google.com/scholar?q=David+Carmel+and+Shaul+Markovitch%2E+Exploration+Strategies+for+Model%2Dbased+Learning+in+Multi%2Dagent+Systems+Autonomous+Agents+and+Multi%2DAgent+Systems+Volume+2%2C+Issue+2+%2C+pp+141%2D172)
24. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, [The Arcade Learning Environment: An Evaluation Platform for General Agents](https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/). JAIR. Volume 47, p.235-279. June 2013.
25. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. Trust Region Policy Optimization Arxiv preprint 1502.05477. A PPENDIX 8.1 O N AUTO ENCODER LAYER SELECTION [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Sergey+Levine%2C+Philipp+Moritz%2C+Michael+I%2E+Jordan%2C+Pieter+Abbeel%2E+Trust+Region+Policy+Optimization+Arxiv+preprint+1502%2E05477%2E+A+PPENDIX+8%2E1+O+N+AUTO+ENCODER+LAYER+SELECTION)

*Updated 3 November 2026*