---
title: "Exploration by Random Network Distillation — Burda 2018"
description: "The authors propose Random Network Distillation (RND). 57 references, cited by 67 papers here."
authors: ["Yuri Burda", "Harrison Edwards", "Amos Storkey", "Oleg Klimov"]
year: 2018
venue: "arXiv:1810.12894"
source: "https://arxiv.org/abs/1810.12894"
doi: "10.48550/arxiv.1810.12894"
arxiv: "1810.12894"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/burda-2018-exploration-random-network-distillation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/burda-2018-exploration-random-network-distillation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Exploration by Random Network Distillation

# Exploration by Random Network Distillation

**Authors:** [Yuri Burda](https://md.research.elamur.ai/free/en/authors/yuri-burda/), Harrison Edwards, [Amos Storkey](https://md.research.elamur.ai/free/en/authors/amos-storkey/) and Oleg Klimov · **Year:** 2018 · **Venue:** arXiv:1810.12894 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1810.12894) · [PDF](https://arxiv.org/pdf/1810.12894) · [DOI 10.48550/arxiv.1810.12894](https://doi.org/10.48550/arxiv.1810.12894) · [arXiv 1810.12894](https://arxiv.org/abs/1810.12894) · [OpenAlex](https://openalex.org/W2899205164)

**Citations:** 235 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/burda-2018-exploration-random-network-distillation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Deep reinforcement learning agents often fail when rewards are sparse. Many existing curiosity-style or count-style exploration techniques are also hard to run across many parallel environments. The authors propose Random Network Distillation (RND). A fixed, randomly initialized target network maps each observation to an embedding, and a separate predictor network learns to reproduce that embedding on states the agent has visited. The predictor's squared error becomes an intrinsic reward, which stays large for unfamiliar observations. Because the target is deterministic and lies within the predictor's model class, the bonus avoids the "noisy-TV" trap that affects forward-dynamics curiosity. The authors also modify PPO with separate value heads for intrinsic and extrinsic rewards. This allows different discount factors and lets the novelty stream be treated as non-episodic. On hard-exploration Atari games, the agent beats average human scores on Montezuma's Revenge without demonstrations or emulator-state access, and it occasionally clears the first level. It also sets new state of the art on Venture and Gravitar. The method costs only one forward pass per batch and scales easily.

## Key points
- The bonus is the MSE between a trained predictor and a frozen random target network. A toy experiment on MNIST shows that this error falls as a class becomes more frequent in training, which supports its use as a novelty signal.
- With extrinsic reward ignored entirely, an RND-driven agent still reliably explores over half the rooms of Montezuma's Revenge. The best combined agent often reaches 22 of the 24 first-level rooms.
- The dual-head PPO variant uses γ_E = 0.999 for extrinsic and γ_I = 0.99 for intrinsic returns. Default runs use 128 parallel environments, sticky actions with probability 0.25, and up to 18K frames per episode.
- Observations fed to the target and predictor networks are normalized and clipped to [−5, 5], and intrinsic rewards are normalized. When scaling to 256 or 1,024 environments, predictor training data is subsampled so that its effective batch size stays constant.
- An ablation replaces RND with an autoencoder reconstruction-error bonus. That variant underperforms RND but still beats plain PPO. CNN-based RND clearly outperforms PPO on Montezuma's Revenge, Private Eye, and Venture.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 381 works · 80,747 citations
- [OpenAI](https://md.research.elamur.ai/free/en/companies/openai/) — Company, San Francisco, United States · 23 works · 27,874 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 134 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Yuri Burda, Harrison Edwards, Amos Storkey and Oleg Klimov. 2018. Exploration by Random Network Distillation. arXiv:1810.12894.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{burda2018exploration,
  title = {{Exploration by Random Network Distillation}},
  author = {Yuri Burda, Harrison Edwards, Amos Storkey and Oleg Klimov},
  year = {2018},
  doi = {10.48550/arxiv.1810.12894},
  eprint = {1810.12894},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.12894}
}
```

- [← previous A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/chollampatt-2018-multilayer-convolutional-encoder-decoder-neural/)
- [next → Learning Named Entity Tagger using Domain-Specific Dictionary](https://md.research.elamur.ai/free/en/papers/shang-2018-learning-named-entity-tagger-domain/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

57 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv:1703.01732, 2017. [find](https://scholar.google.com/scholar?q=Joshua+Achiam+and+Shankar+Sastry%2E+Surprise%2Dbased+intrinsic+motivation+for+deep+reinforcement+learning%2E+arXiv%3A1703%2E01732%2C+2017%2E)
2. Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299, 2018. — [arXiv:1807.10299](https://arxiv.org/abs/1807.10299) [find](https://scholar.google.com/scholar?q=Joshua+Achiam%2C+Harrison+Edwards%2C+Dario+Amodei%2C+and+Pieter+Abbeel%2E+Variational+option+discovery+algorithms%2E+arXiv+preprint+arXiv%3A1807%2E10299%2C+2018%2E+%E2%80%94+%5BarXiv%3A1807%2E10299%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E10299%29)
3. Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching YouTube. arXiv preprint arXiv:1805.11592, 2018. — [arXiv:1805.11592](https://arxiv.org/abs/1805.11592) [find](https://scholar.google.com/scholar?q=Yusuf+Aytar%2C+Tobias+Pfaff%2C+David+Budden%2C+Tom+Le+Paine%2C+Ziyu+Wang%2C+and+Nando+de+Freitas%2E+Playing+hard+exploration+games+by+watching+YouTube%2E+arXiv+preprint+arXiv%3A1805%2E11592%2C+2018%2E+%E2%80%94+%5BarXiv%3A1805%2E11592%5D%28)
4. Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, 2016. [find](https://scholar.google.com/scholar?q=Marc+Bellemare%2C+Sriram+Srinivasan%2C+Georg+Ostrovski%2C+Tom+Schaul%2C+David+Saxton%2C+and+Remi+Munos%2E+Unifying+count%2Dbased+exploration+and+intrinsic+motivation%2E+In+NIPS%2C+2016%2E)
5. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. arXiv preprint arXiv:1707.06887, 2017. — [arXiv:1707.06887](https://arxiv.org/abs/1707.06887) [find](https://scholar.google.com/scholar?q=Marc+G+Bellemare%2C+Will+Dabney%2C+and+Re%CC%81mi+Munos%2E+A+distributional+perspective+on+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1707%2E06887%2C+2017%2E+%E2%80%94+%5BarXiv%3A1707%2E06887%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E06887%29)
6. Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In arXiv:1808.04355, 2018. [find](https://scholar.google.com/scholar?q=Yuri+Burda%2C+Harri+Edwards%2C+Deepak+Pathak%2C+Amos+Storkey%2C+Trevor+Darrell%2C+and+Alexei+A%2E+Efros%2E+Large%2Dscale+study+of+curiosity%2Ddriven+learning%2E+In+arXiv%3A1808%2E04355%2C+2018%2E)
7. Richard Y Chen, John Schulman, Pieter Abbeel, and Szymon Sidor. UCB and infogain exploration via q-ensembles. arXiv:1706.01502, 2017. [find](https://scholar.google.com/scholar?q=Richard+Y+Chen%2C+John+Schulman%2C+Pieter+Abbeel%2C+and+Szymon+Sidor%2E+UCB+and+infogain+exploration+via+q%2Densembles%2E+arXiv%3A1706%2E01502%2C+2017%2E)
8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.[1078, 2014](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/).
9. Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, and Nando de Freitas. Learning to perform physics experiments via deep reinforcement learning. arXiv preprint arXiv:1611.01843, 2016. — [arXiv:1611.01843](https://arxiv.org/abs/1611.01843) [find](https://scholar.google.com/scholar?q=Misha+Denil%2C+Pulkit+Agrawal%2C+Tejas+D+Kulkarni%2C+Tom+Erez%2C+Peter+Battaglia%2C+and+Nando+de+Freitas%2E+Learning+to+perform+physics+experiments+via+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1611%2E01843)
10. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. IMPALA: Scalable distributed Deep-RL with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018. [find](https://scholar.google.com/scholar?q=Lasse+Espeholt%2C+Hubert+Soyer%2C+Remi+Munos%2C+Karen+Simonyan%2C+Volodymir+Mnih%2C+Tom+Ward%2C+Yotam+Doron%2C+Vlad+Firoiu%2C+Tim+Harley%2C+Iain+Dunning%2C+et+al%2E+IMPALA%3A+Scalable+distributed+Deep%2DRL+with+importance+weig)
11. Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. [Diversity is all you need: Learning skills without a reward function](https://md.research.elamur.ai/free/en/papers/eysenbach-2018-diversity-all-need-learning-skills/). arXiv preprint, 2018.
12. Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. arXiv:1706.10295, 2017. [find](https://scholar.google.com/scholar?q=Meire+Fortunato%2C+Mohammad+Gheshlaghi+Azar%2C+Bilal+Piot%2C+Jacob+Menick%2C+Ian+Osband%2C+Alex+Graves%2C+Vlad+Mnih%2C+Remi+Munos%2C+Demis+Hassabis%2C+Olivier+Pietquin%2C+Charles+Blundell%2C+and+Shane+Legg%2E+Noisy+networks+)
13. Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Lior+Fox%2C+Leshem+Choshen%2C+and+Yonatan+Loewenstein%2E+Dora+the+explorer%3A+Directed+outreaching+reinforcement+action%2Dselection%2E+International+Conference+on+Learning+Representations%2C+2018%2E)
14. Justin Fu, John D Co-Reyes, and Sergey Levine. EX2: Exploration with exemplar models for deep reinforcement learning. NIPS, 2017. [find](https://scholar.google.com/scholar?q=Justin+Fu%2C+John+D+Co%2DReyes%2C+and+Sergey+Levine%2E+EX2%3A+Exploration+with+exemplar+models+for+deep+reinforcement+learning%2E+NIPS%2C+2017%2E)
15. Michał Garmulewicz, Henryk Michalewski, and Piotr Miłoś. Expert-augmented actor-critic for vizdoom and montezumas revenge. arXiv preprint arXiv:1809.03447, 2018. — [arXiv:1809.03447](https://arxiv.org/abs/1809.03447) [find](https://scholar.google.com/scholar?q=Micha%C5%82+Garmulewicz%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+Expert%2Daugmented+actor%2Dcritic+for+vizdoom+and+montezumas+revenge%2E+arXiv+preprint+arXiv%3A1809%2E03447%2C+2018%2E+%E2%80%94+%5BarXiv%3A1809%2E03447%5D%28https%3A%2F%2Farxiv%2Eor)
16. Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. ICLR Workshop, 2017. [find](https://scholar.google.com/scholar?q=Karol+Gregor%2C+Danilo+Jimenez+Rezende%2C+and+Daan+Wierstra%2E+Variational+intrinsic+control%2E+ICLR+Workshop%2C+2017%2E)
17. Nick Haber, Damian Mrowca, Li Fei-Fei, and Daniel LK Yamins. Learning to play with intrinsically-motivated self-aware agents. arXiv preprint arXiv:1802.07442, 2018. — [arXiv:1802.07442](https://arxiv.org/abs/1802.07442) [find](https://scholar.google.com/scholar?q=Nick+Haber%2C+Damian+Mrowca%2C+Li+Fei%2DFei%2C+and+Daniel+LK+Yamins%2E+Learning+to+play+with+intrinsically%2Dmotivated+self%2Daware+agents%2E+arXiv+preprint+arXiv%3A1802%2E07442%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E07442%5D%28https%3A%2F%2Farxiv%2E)
18. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298, 2017. [find](https://scholar.google.com/scholar?q=Matteo+Hessel%2C+Joseph+Modayil%2C+Hado+Van+Hasselt%2C+Tom+Schaul%2C+Georg+Ostrovski%2C+Will+Dabney%2C+Dan+Horgan%2C+Bilal+Piot%2C+Mohammad+Azar%2C+and+David+Silver%2E+Rainbow%3A+Combining+improvements+in+deep+reinforcemen)
19. Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018. — [arXiv:1803.00933](https://arxiv.org/abs/1803.00933) [find](https://scholar.google.com/scholar?q=Dan+Horgan%2C+John+Quan%2C+David+Budden%2C+Gabriel+Barth%2DMaron%2C+Matteo+Hessel%2C+Hado+Van+Hasselt%2C+and+David+Silver%2E+Distributed+prioritized+experience+replay%2E+arXiv+preprint+arXiv%3A1803%2E00933%2C+2018%2E+%E2%80%94+%5BarXiv%3A)
20. Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational information maximizing exploration. In NIPS, 2016. [find](https://scholar.google.com/scholar?q=Rein+Houthooft%2C+Xi+Chen%2C+Yan+Duan%2C+John+Schulman%2C+Filip+De+Turck%2C+and+Pieter+Abbeel%2E+VIME%3A+Variational+information+maximizing+exploration%2E+In+NIPS%2C+2016%2E)
21. Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. What is the best multi-stage architecture for object recognition? In Computer Vision, 2009 IEEE 12th International Conference on, pp. 2146–2153. [find](https://scholar.google.com/scholar?q=Kevin+Jarrett%2C+Koray+Kavukcuoglu%2C+Yann+LeCun%2C+et+al%2E+What+is+the+best+multi%2Dstage+architecture+for+object+recognition%3F+In+Computer+Vision%2C+2009+IEEE+12th+International+Conference+on%2C+pp%2E+2146%E2%80%932153%2E)
22. Diederik Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). ICLR, 2015.
23. Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675–3683, 2016. [find](https://scholar.google.com/scholar?q=Tejas+D+Kulkarni%2C+Karthik+Narasimhan%2C+Ardavan+Saeedi%2C+and+Josh+Tenenbaum%2E+Hierarchical+deep+reinforcement+learning%3A+Integrating+temporal+abstraction+and+intrinsic+motivation%2E+In+Advances+in+neural+inf)
24. Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre-Yves Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. In NIPS, 2012. [find](https://scholar.google.com/scholar?q=Manuel+Lopes%2C+Tobias+Lang%2C+Marc+Toussaint%2C+and+Pierre%2DYves+Oudeyer%2E+Exploration+in+model%2Dbased+reinforcement+learning+by+empirically+estimating+learning+progress%2E+In+NIPS%2C+2012%2E)
25. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. [Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents](https://md.research.elamur.ai/free/en/papers/machado-2017-revisiting-arcade-learning-environment-evaluation/). arXiv preprint arXiv:1709.06009, 2017.
26. Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. arXiv preprint arXiv:1807.11622, 2018. — [arXiv:1807.11622](https://arxiv.org/abs/1807.11622) [find](https://scholar.google.com/scholar?q=Marlos+C+Machado%2C+Marc+G+Bellemare%2C+and+Michael+Bowling%2E+Count%2Dbased+exploration+with+the+successor+representation%2E+arXiv+preprint+arXiv%3A1807%2E11622%2C+2018%2E+%E2%80%94+%5BarXiv%3A1807%2E11622%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F18)
27. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Alex+Graves%2C+Ioannis+Antonoglou%2C+Daan+Wierstra%2C+and+Martin+Riedmiller%2E+Playing+Atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312%2E5602%2C+)
28. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Andrei+A%2E+Rusu%2C+Joel+Veness%2C+Marc+G%2E+Bellemare%2C+Alex+Graves%2C+Martin+Riedmiller%2C+Andreas+K%2E+Fidjeland%2C+Georg+Ostrovski%2C+Stig+Petersen%2C+Charles+Beattie%2C+)
29. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Adria+Puigdomenech+Badia%2C+Mehdi+Mirza%2C+Alex+Graves%2C+Timothy+Lillicrap%2C+Tim+Harley%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+In+ICML%2C+2)
30. Brendan O’Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty Bellman equation and exploration. arXiv preprint arXiv:1709.05380, 2017. — [arXiv:1709.05380](https://arxiv.org/abs/1709.05380) [find](https://scholar.google.com/scholar?q=Brendan+O%E2%80%99Donoghue%2C+Ian+Osband%2C+Remi+Munos%2C+and+Volodymyr+Mnih%2E+The+uncertainty+Bellman+equation+and+exploration%2E+arXiv+preprint+arXiv%3A1709%2E05380%2C+2017%2E+%E2%80%94+%5BarXiv%3A1709%2E05380%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1709)
31. Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint arXiv:1806.05635, 2018. — [arXiv:1806.05635](https://arxiv.org/abs/1806.05635) [find](https://scholar.google.com/scholar?q=Junhyuk+Oh%2C+Yijie+Guo%2C+Satinder+Singh%2C+and+Honglak+Lee%2E+Self%2Dimitation+learning%2E+arXiv+preprint+arXiv%3A1806%2E05635%2C+2018%2E+%E2%80%94+%5BarXiv%3A1806%2E05635%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1806%2E05635%29)
32. OpenAI. OpenAI Five. [https://blog.openai.com/openai-five/](https://blog.openai.com/openai-five/), 2018. [↗](https://blog.openai.com/openai-five/)
33. OpenAI, :, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning Dexterous In-Hand Manipulation. ArXiv e-prints, August 2018. [find](https://scholar.google.com/scholar?q=OpenAI%2C+%3A%2C+M%2E+Andrychowicz%2C+B%2E+Baker%2C+M%2E+Chociej%2C+R%2E+Jozefowicz%2C+B%2E+McGrew%2C+J%2E+Pachocki%2C+A%2E+Petron%2C+M%2E+Plappert%2C+G%2E+Powell%2C+A%2E+Ray%2C+J%2E+Schneider%2C+S%2E+Sidor%2C+J%2E+Tobin%2C+P%2E+Welinder%2C+L%2E+Weng%2C+and+W%2E+Zarem)
34. Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. [Deep exploration via bootstrapped DQN](https://md.research.elamur.ai/free/en/papers/osband-2016-deep-exploration-bootstrapped-dqn/). In NIPS, 2016.
35. Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv:1806.03335, 2018. — [arXiv:1806.03335](https://arxiv.org/abs/1806.03335) [find](https://scholar.google.com/scholar?q=Ian+Osband%2C+John+Aslanides%2C+and+Albin+Cassirer%2E+Randomized+prior+functions+for+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1806%2E03335%2C+2018%2E+%E2%80%94+%5BarXiv%3A1806%2E03335%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1806%2E03335)
36. Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and Rémi Munos. Count-based exploration with neural density models. arXiv:1703.01310, 2017. — [arXiv:1703.01310](https://arxiv.org/abs/1703.01310) [find](https://scholar.google.com/scholar?q=Georg+Ostrovski%2C+Marc+G+Bellemare%2C+Aaron+van+den+Oord%2C+and+Re%CC%81mi+Munos%2E+Count%2Dbased+exploration+with+neural+density+models%2E+arXiv%3A1703%2E01310%2C+2017%2E+%E2%80%94+%5BarXiv%3A1703%2E01310%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1703%2E0131)
37. Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and Rémi Munos. Count-based exploration with neural density models. International Conference for Machine Learning, 2018. [find](https://scholar.google.com/scholar?q=Georg+Ostrovski%2C+Marc+G+Bellemare%2C+Aaron+van+den+Oord%2C+and+Re%CC%81mi+Munos%2E+Count%2Dbased+exploration+with+neural+density+models%2E+International+Conference+for+Machine+Learning%2C+2018%2E)
38. Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. Evolutionary Computation, 2007. [find](https://scholar.google.com/scholar?q=Pierre%2DYves+Oudeyer%2C+Frdric+Kaplan%2C+and+Verena+V+Hafner%2E+Intrinsic+motivation+systems+for+autonomous+mental+development%2E+Evolutionary+Computation%2C+2007%2E)
39. Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017. [find](https://scholar.google.com/scholar?q=Deepak+Pathak%2C+Pulkit+Agrawal%2C+Alexei+A%2E+Efros%2C+and+Trevor+Darrell%2E+Curiosity%2Ddriven+exploration+by+self%2Dsupervised+prediction%2E+In+ICML%2C+2017%2E)
40. Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv:1706.01905, 2017. [find](https://scholar.google.com/scholar?q=Matthias+Plappert%2C+Rein+Houthooft%2C+Prafulla+Dhariwal%2C+Szymon+Sidor%2C+Richard+Y+Chen%2C+Xi+Chen%2C+Tamim+Asfour%2C+Pieter+Abbeel%2C+and+Marcin+Andrychowicz%2E+Parameter+space+noise+for+exploration%2E+arXiv%3A1706%2E019)
41. Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Večerı́k, et al. Observe and look further: Achieving consistent performance on Atari. arXiv preprint arXiv:1805.11593, 2018. — [arXiv:1805.11593](https://arxiv.org/abs/1805.11593) [find](https://scholar.google.com/scholar?q=Tobias+Pohlen%2C+Bilal+Piot%2C+Todd+Hester%2C+Mohammad+Gheshlaghi+Azar%2C+Dan+Horgan%2C+David+Budden%2C+Gabriel+Barth%2DMaron%2C+Hado+van+Hasselt%2C+John+Quan%2C+Mel+Vec%CC%8Cer%C4%B1%CC%81k%2C+et+al%2E+Observe+and+look+further%3A+Achieving+)
42. Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep RL for model-based control. arXiv preprint arXiv:1802.09081, 2018. — [arXiv:1802.09081](https://arxiv.org/abs/1802.09081) [find](https://scholar.google.com/scholar?q=Vitchyr+Pong%2C+Shixiang+Gu%2C+Murtaza+Dalal%2C+and+Sergey+Levine%2E+Temporal+difference+models%3A+Model%2Dfree+deep+RL+for+model%2Dbased+control%2E+arXiv+preprint+arXiv%3A1802%2E09081%2C+2018%2E+%E2%80%94+%5BarXiv%3A1802%2E09081%5D%28https%3A%2F)
43. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, pp. 1177–1184, 2008. [find](https://scholar.google.com/scholar?q=Ali+Rahimi+and+Benjamin+Recht%2E+Random+features+for+large%2Dscale+kernel+machines%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+1177%E2%80%931184%2C+2008%2E)
44. Tim Salimans and Richard Chen. Learning Montezuma’s Revenge from a single demonstration. [https://blog.openai.com/learning-montezumas-revenge-from-a-single-demonstration/](https://blog.openai.com/learning-montezumas-revenge-from-a-single-demonstration/), 2018. [↗](https://blog.openai.com/learning-montezumas-revenge-from-a-single-demonstration/)
45. Andrew M Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. On random weights and unsupervised feature learning. In ICML, pp. 1089–1096, 2011. [find](https://scholar.google.com/scholar?q=Andrew+M+Saxe%2C+Pang+Wei+Koh%2C+Zhenghao+Chen%2C+Maneesh+Bhand%2C+Bipin+Suresh%2C+and+Andrew+Y+Ng%2E+On+random+weights+and+unsupervised+feature+learning%2E+In+ICML%2C+pp%2E+1089%E2%80%931096%2C+2011%2E)
46. Jürgen Schmidhuber. Curious model-building control systems. In Neural Networks, 1991. 1991 IEEE International Joint Conference on, pp. 1458–1463. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+Curious+model%2Dbuilding+control+systems%2E+In+Neural+Networks%2C+1991%2E+1991+IEEE+International+Joint+Conference+on%2C+pp%2E+1458%E2%80%931463%2E)
47. Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the First International Conference on Simulation of Adaptive Behavior, 1991b. [find](https://scholar.google.com/scholar?q=Ju%CC%88rgen+Schmidhuber%2E+A+possibility+for+implementing+curiosity+and+boredom+in+model%2Dbuilding+neural+controllers%2E+In+Proceedings+of+the+First+International+Conference+on+Simulation+of+Adaptive+Behavior%2C)
48. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
49. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukc… [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J%2E+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+van+den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+Sander+Dieleman%2C+Dominik+)
50. Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. NIPS Workshop, 2015. [find](https://scholar.google.com/scholar?q=Bradly+C+Stadie%2C+Sergey+Levine%2C+and+Pieter+Abbeel%2E+Incentivizing+exploration+in+reinforcement+learning+with+deep+predictive+models%2E+NIPS+Workshop%2C+2015%2E)
51. Christopher Stanton and Jeff Clune. Deep curiosity search: Intra-life exploration improves performance on challenging deep reinforcement learning problems. arXiv preprint arXiv:1806.00553, 2018. — [arXiv:1806.00553](https://arxiv.org/abs/1806.00553) [find](https://scholar.google.com/scholar?q=Christopher+Stanton+and+Jeff+Clune%2E+Deep+curiosity+search%3A+Intra%2Dlife+exploration+improves+performance+on+challenging+deep+reinforcement+learning+problems%2E+arXiv+preprint+arXiv%3A1806%2E00553%2C+2018%2E+%E2%80%94+%5Bar)
52. Susanne Still and Doina Precup. An information-theoretic approach to curiosity-driven reinforcement learning. Theory in Biosciences, 2012. [find](https://scholar.google.com/scholar?q=Susanne+Still+and+Doina+Precup%2E+An+information%2Dtheoretic+approach+to+curiosity%2Ddriven+reinforcement+learning%2E+Theory+in+Biosciences%2C+2012%2E)
53. Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309–1331, 2008. [find](https://scholar.google.com/scholar?q=Alexander+L+Strehl+and+Michael+L+Littman%2E+An+analysis+of+model%2Dbased+interval+estimation+for+markov+decision+processes%2E+Journal+of+Computer+and+System+Sciences%2C+74%288%29%3A1309%E2%80%931331%2C+2008%2E)
54. Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Ilya+Kostrikov%2C+Arthur+Szlam%2C+and+Rob+Fergus%2E+Intrinsic+motivation+and+automatic+curricula+via+asymmetric+self%2Dplay%2E+In+ICLR%2C+2018%2E)
55. Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. \# exploration: A study of count-based exploration for deep reinforcement learning. In NIPS, 2017. [find](https://scholar.google.com/scholar?q=Haoran+Tang%2C+Rein+Houthooft%2C+Davis+Foote%2C+Adam+Stooke%2C+Xi+Chen%2C+Yan+Duan%2C+John+Schulman%2C+Filip+DeTurck%2C+and+Pieter+Abbeel%2E+%5C%23+exploration%3A+A+study+of+count%2Dbased+exploration+for+deep+reinforcement+lea)
56. Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1476–1483, 2015. [find](https://scholar.google.com/scholar?q=Zichao+Yang%2C+Marcin+Moczulski%2C+Misha+Denil%2C+Nando+de+Freitas%2C+Alex+Smola%2C+Le+Song%2C+and+Ziyu+Wang%2E+Deep+fried+convnets%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2C+pp%2E+1476%E2%80%93)
57. Barret Zoph and Quoc V Le. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). arXiv preprint arXiv:1611.01578, 2016.

*Updated 2 November 2026*