---
title: "Revisiting the Arcade Learning Environment: Evaluation"
description: "This article reviews how the community has used the ALE, shows how evaluation protocols have… 79 references, cited by 40 papers here."
authors: ["Marlos C. Machado", "Marc G. Bellemare", "Erik Talvitie", "Joel Veness", "Matthew Hausknecht", "Michael Bowling"]
year: 2017
venue: "arXiv:1709.06009"
source: "https://arxiv.org/abs/1709.06009"
doi: "10.48550/arxiv.1709.06009"
arxiv: "1709.06009"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/machado-2017-revisiting-arcade-learning-environment-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/machado-2017-revisiting-arcade-learning-environment-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

# Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

**Authors:** Marlos C. Machado, [Marc G. Bellemare](https://md.research.elamur.ai/free/en/authors/marc-g-bellemare/), Erik Talvitie, [Joel Veness](https://md.research.elamur.ai/free/en/authors/joel-veness/), [Matthew Hausknecht](https://md.research.elamur.ai/free/en/authors/matthew-hausknecht/) and [Michael Bowling](https://md.research.elamur.ai/free/en/authors/michael-bowling/) · **Year:** 2017 · **Venue:** [arXiv:1709.06009](https://md.research.elamur.ai/free/en/venues/journal-of-artificial-intelligence-research/) ([Journal of Artificial Intelligence Research 2017](https://md.research.elamur.ai/free/en/venues/journal-of-artificial-intelligence-research/2017/)) · **Publisher:** [AI Access Foundation](https://md.research.elamur.ai/free/en/publishers/ai-access-foundation/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1709.06009) · [PDF](https://arxiv.org/pdf/1709.06009) · [DOI 10.48550/arxiv.1709.06009](https://doi.org/10.48550/arxiv.1709.06009) · [arXiv 1709.06009](https://arxiv.org/abs/1709.06009) · [OpenAlex](https://openalex.org/W2754879180)

**Citations:** 353 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/machado-2017-revisiting-arcade-learning-environment-evaluation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The Arcade Learning Environment (ALE) is a popular testbed for general AI agents that must play many Atari 2600 games without game-specific knowledge. As its use grew, research groups adopted different and often incompatible evaluation setups, which made published results hard to compare. This article reviews how the community has used the ALE, shows how evaluation protocols have diverged and been mixed up, and proposes best practices for the standard reinforcement learning setting. The authors also respond to criticism that the emulator is fully deterministic. They release a new ALE version with a stochasticity mechanism called sticky actions and support for alternative game modes and difficulty levels. Following their own recommendations, they report new benchmark results for Sarsa(λ) with Blob-PROST features (linear) and DQN (neural). Performance is reported at several stages of training to encourage attention to sample efficiency. Finally, they revisit the challenges from the original ALE paper and summarize the state of the art. They name five problems that remain open, offering guidance for future work on this and similar general-competency benchmarks.

## Key points
- In the ALE, an agent sees a 210 × 160 screen image and/or the 1024-bit RAM state. One frame lasts 1/60 s, and the underlying emulator is deterministic.
- Common preprocessing steps include frame skipping (typically k = 4 or 5), color averaging or frame pooling, and frame stacking. Rewards are often normalized, for example by clipping them to their sign as in DQN.
- The new ALE release adds sticky actions to inject randomness. It also adds "flavors," meaning mode–difficulty pairs such as the different modes of Freeway, which create many closely related environments suited to transfer learning.
- Benchmarks compare linear Sarsa(λ) + Blob-PROST against DQN and report results at several points during learning, not only the final score.
- The five open problems are representation learning, planning and model learning, exploration, transfer learning, and off-policy learning. For off-policy learning, the authors note that methods like Q-learning and GQ(λ) tend to be unstable in the ALE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 72 works · 84,326 citations
- [University of Alberta](https://md.research.elamur.ai/free/en/universities/university-of-alberta/) — University, Edmonton, Canada · 51 works · 5,017 citations

Also on the byline: [Franklin & Marshall College](https://ror.org/04fp4ps48).

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht and Michael Bowling. 2017. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. arXiv:1709.06009.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{machado2017revisiting,
  title = {{Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents}},
  author = {Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht and Michael Bowling},
  year = {2017},
  doi = {10.48550/arxiv.1709.06009},
  eprint = {1709.06009},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.06009}
}
```

- [← previous URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors](https://md.research.elamur.ai/free/en/papers/littell-2017-uriel-lang2vec-representing-languages-as/)
- [next → Deep Semantic Role Labeling with Self-Attention](https://md.research.elamur.ai/free/en/papers/tan-2017-deep-semantic-role-labeling-self/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/)

## References

79 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Albrecht, S. V., L., J. C., Buckeridge, D. L., Botea, A., Caragea, C., Chi, C., Damoulas, T., Dilkina, B. N., Eaton, E., Fazli, P., Ganzfried, S., Lindauer, M. T., Machado, M. C., Malitsky, Y., Marcus, G., Meijer, S., Rossi, F., Shaban-Nejad, A., Thiebaux, S., Veloso, M. M., Walsh, T., Wang, C., Zha… [find](https://scholar.google.com/scholar?q=Albrecht%2C+S%2E+V%2E%2C+L%2E%2C+J%2E+C%2E%2C+Buckeridge%2C+D%2E+L%2E%2C+Botea%2C+A%2E%2C+Caragea%2C+C%2E%2C+Chi%2C+C%2E%2C+Damoulas%2C+T%2E%2C+Dilkina%2C+B%2E+N%2E%2C+Eaton%2C+E%2E%2C+Fazli%2C+P%2E%2C+Ganzfried%2C+S%2E%2C+Lindauer%2C+M%2E+T%2E%2C+Machado%2C+M%2E+C%2E%2C+Malitsky%2C+Y%2E%2C+Marcus)
2. Barto, A. G. (2013). Intrinsic Motivation and Reinforcement Learning. In Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 17–47. Springer. [find](https://scholar.google.com/scholar?q=Barto%2C+A%2E+G%2E+%282013%29%2E+Intrinsic+Motivation+and+Reinforcement+Learning%2E+In+Intrinsically+Motivated+Learning+in+Natural+and+Artificial+Systems%2C+pp%2E+17%E2%80%9347%2E+Springer%2E)
3. Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., Schrittwieser, J., Anderson, K., York, S., Cant, M., Cain, A., Bolton, A., Gaffney, S., King, H., Hassabis, D., Legg, S., & Petersen, S. (2016). [DeepMind Lab](https://md.research.elamur.ai/free/en/papers/beattie-2016-deepmind-lab/). CoRR, abs…
4. Bellemare, M., Veness, J., & Bowling, M. (2013). Bayesian Learning of Recursively Factored Environments. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1211–1219. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E%2C+Veness%2C+J%2E%2C+%26+Bowling%2C+M%2E+%282013%29%2E+Bayesian+Learning+of+Recursively+Factored+Environments%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2C+pp%2E+1211%E2%80%931219%2E)
5. Bellemare, M. G., Dabney, W., & Munos, R. (2017). A Distributional Perspective on Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Dabney%2C+W%2E%2C+%26+Munos%2C+R%2E+%282017%29%2E+A+Distributional+Perspective+on+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2E)
6. Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013). [The Arcade Learning Environment: An Evaluation Platform for General Agents](https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/). Journal of Artificial Intelligence Research, 47, 253–279.
7. Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2015). The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract). In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 4148–4152. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Naddaf%2C+Y%2E%2C+Veness%2C+J%2E%2C+%26+Bowling%2C+M%2E+%282015%29%2E+The+Arcade+Learning+Environment%3A+An+Evaluation+Platform+for+General+Agents+%28Extended+Abstract%29%2E+In+Proceedings+of+the+International+Join)
8. Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P. S., & Munos, R. (2016a). Increasing the Action Gap: New Operators for Reinforcement Learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 1476–1483. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Ostrovski%2C+G%2E%2C+Guez%2C+A%2E%2C+Thomas%2C+P%2E+S%2E%2C+%26+Munos%2C+R%2E+%282016a%29%2E+Increasing+the+Action+Gap%3A+New+Operators+for+Reinforcement+Learning%2E+In+Proceedings+of+the+Conference+on+Artificial+Intel)
9. Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016b). Unifying Count-Based Exploration and Intrinsic Motivation. In Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Srinivasan%2C+S%2E%2C+Ostrovski%2C+G%2E%2C+Schaul%2C+T%2E%2C+Saxton%2C+D%2E%2C+%26+Munos%2C+R%2E+%282016b%29%2E+Unifying+Count%2DBased+Exploration+and+Intrinsic+Motivation%2E+In+Advances+in+Neural+Information+Processing+Sy)
10. Bellemare, M. G., Veness, J., & Bowling, M. (2012a). Investigating Contingency Awareness using Atari 2600 Games. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 864–871. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Veness%2C+J%2E%2C+%26+Bowling%2C+M%2E+%282012a%29%2E+Investigating+Contingency+Awareness+using+Atari+2600+Games%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2C+pp%2E+864%E2%80%93871%2E)
11. Bellemare, M. G., Veness, J., & Bowling, M. (2012b). Sketch-Based Linear Value Function Approximation. In Advances in Neural Information Processing Systems (NIPS), pp. 2222–2230. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Veness%2C+J%2E%2C+%26+Bowling%2C+M%2E+%282012b%29%2E+Sketch%2DBased+Linear+Value+Function+Approximation%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pp%2E+2222%E2%80%932230%2E)
12. Bellemare, M. G., Veness, J., & Talvitie, E. (2014). Skip Context Tree Switching. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1458– 1466. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Veness%2C+J%2E%2C+%26+Talvitie%2C+E%2E+%282014%29%2E+Skip+Context+Tree+Switching%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2C+pp%2E+1458%E2%80%93+1466%2E)
13. Bellman, R. E. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ. [find](https://scholar.google.com/scholar?q=Bellman%2C+R%2E+E%2E+%281957%29%2E+Dynamic+Programming%2E+Princeton+University+Press%2C+Princeton%2C+NJ%2E)
14. Brafman, R. I., & Tennenholtz, M. (2002). R-MAX - A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3, 213–231. [find](https://scholar.google.com/scholar?q=Brafman%2C+R%2E+I%2E%2C+%26+Tennenholtz%2C+M%2E+%282002%29%2E+R%2DMAX+%2D+A+General+Polynomial+Time+Algorithm+for+Near%2DOptimal+Reinforcement+Learning%2E+Journal+of+Machine+Learning+Research%2C+3%2C+213%E2%80%93231%2E)
15. Braylan, A., Hollenbeck, M., Meyerson, E., & Miikkulainen, R. (2015). Frame Skip is a Powerful Parameter for Learning to Play Atari. In AAAI Workshop on Learning for General Competency in Video Games. [find](https://scholar.google.com/scholar?q=Braylan%2C+A%2E%2C+Hollenbeck%2C+M%2E%2C+Meyerson%2C+E%2E%2C+%26+Miikkulainen%2C+R%2E+%282015%29%2E+Frame+Skip+is+a+Powerful+Parameter+for+Learning+to+Play+Atari%2E+In+AAAI+Workshop+on+Learning+for+General+Competency+in+Video+Games%2E)
16. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). [OpenAI Gym](https://md.research.elamur.ai/free/en/papers/nair-2015-massively-parallel-methods-deep-reinforcement/). CoRR, abs/1507.04296.
17. Campbell, M., Jr., A. J. H., & Hsu, F. (2002). Deep Blue. Artificial Intelligence, 134 (1-2), 57–83. [find](https://scholar.google.com/scholar?q=Campbell%2C+M%2E%2C+Jr%2E%2C+A%2E+J%2E+H%2E%2C+%26+Hsu%2C+F%2E+%282002%29%2E+Deep+Blue%2E+Artificial+Intelligence%2C+134+%281%2D2%29%2C+57%E2%80%9383%2E)
18. Chiappa, S., Racaniere, S., Wierstra, D., & Mohamed, S. (2017). Recurrent Environment Simulators. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Chiappa%2C+S%2E%2C+Racaniere%2C+S%2E%2C+Wierstra%2C+D%2E%2C+%26+Mohamed%2C+S%2E+%282017%29%2E+Recurrent+Environment+Simulators%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
19. Defazio, A., & Graepel, T. (2014). A Comparison of Learning Algorithms on the Arcade Learning Environment. CoRR, abs/1410.8620. — [arXiv:1410.8620](https://arxiv.org/abs/1410.8620) [find](https://scholar.google.com/scholar?q=Defazio%2C+A%2E%2C+%26+Graepel%2C+T%2E+%282014%29%2E+A+Comparison+of+Learning+Algorithms+on+the+Arcade+Learning+Environment%2E+CoRR%2C+abs%2F1410%2E8620%2E+%E2%80%94+%5BarXiv%3A1410%2E8620%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1410%2E8620%29)
20. Even-Dar, E., Kakade, S. M., & Mansour, Y. (2005). Reinforcement Learning in POMDPs Without Resets. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 690–695. [find](https://scholar.google.com/scholar?q=Even%2DDar%2C+E%2E%2C+Kakade%2C+S%2E+M%2E%2C+%26+Mansour%2C+Y%2E+%282005%29%2E+Reinforcement+Learning+in+POMDPs+Without+Resets%2E+In+Proceedings+of+the+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%29%2C+pp%2E+690%E2%80%93695)
21. Farahmand, A.-M., Barreto, A., & Nikovski, D. (2017). Value-Aware Loss Function for Model-based Reinforcement Learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). [find](https://scholar.google.com/scholar?q=Farahmand%2C+A%2E%2DM%2E%2C+Barreto%2C+A%2E%2C+%26+Nikovski%2C+D%2E+%282017%29%2E+Value%2DAware+Loss+Function+for+Model%2Dbased+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Conference+on+Artificial+Intelligence+and+St)
22. Guo, X., Singh, S., Lee, H., Lewis, R. L., & Wang, X. (2014). Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning. In Advances in Neural Information Processing Systems (NIPS), pp. 3338–3346. [find](https://scholar.google.com/scholar?q=Guo%2C+X%2E%2C+Singh%2C+S%2E%2C+Lee%2C+H%2E%2C+Lewis%2C+R%2E+L%2E%2C+%26+Wang%2C+X%2E+%282014%29%2E+Deep+Learning+for+Real%2DTime+Atari+Game+Play+Using+Offline+Monte%2DCarlo+Tree+Search+Planning%2E+In+Advances+in+Neural+Information+Processing+S)
23. Harutyunyan, A., Bellemare, M. G., Stepleton, T., & Munos, R. (2016). Q(λ) with Off-Policy Corrections. In Proceedings of the International Conference on Algorithmic Learning Theory (ALT). [find](https://scholar.google.com/scholar?q=Harutyunyan%2C+A%2E%2C+Bellemare%2C+M%2E+G%2E%2C+Stepleton%2C+T%2E%2C+%26+Munos%2C+R%2E+%282016%29%2E+Q%28%CE%BB%29+with+Off%2DPolicy+Corrections%2E+In+Proceedings+of+the+International+Conference+on+Algorithmic+Learning+Theory+%28ALT%29%2E)
24. Hausknecht, M., & Stone, P. (2015). The Impact of Determinism on Learning Atari 2600 Games. In AAAI Workshop on Learning for General Competency in Video Games. [find](https://scholar.google.com/scholar?q=Hausknecht%2C+M%2E%2C+%26+Stone%2C+P%2E+%282015%29%2E+The+Impact+of+Determinism+on+Learning+Atari+2600+Games%2E+In+AAAI+Workshop+on+Learning+for+General+Competency+in+Video+Games%2E)
25. Hausknecht, M. J., Lehman, J., Miikkulainen, R., & Stone, P. (2014). A Neuroevolution Approach to General Atari Game Playing. IEEE Transactions on Computational Intelligence and AI in Games, 6 (4), 355–366. [find](https://scholar.google.com/scholar?q=Hausknecht%2C+M%2E+J%2E%2C+Lehman%2C+J%2E%2C+Miikkulainen%2C+R%2E%2C+%26+Stone%2C+P%2E+%282014%29%2E+A+Neuroevolution+Approach+to+General+Atari+Game+Playing%2E+IEEE+Transactions+on+Computational+Intelligence+and+AI+in+Games%2C+6+%284%29%2C+35)
26. Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Sendonaris, A., Dulac-Arnold, G., Osband, I., Agapiou, J., et al. (2017). Learning from Demonstrations for Real World Reinforcement Learning. CoRR, abs/1704.03732. — [arXiv:1704.03732](https://arxiv.org/abs/1704.03732) [find](https://scholar.google.com/scholar?q=Hester%2C+T%2E%2C+Vecerik%2C+M%2E%2C+Pietquin%2C+O%2E%2C+Lanctot%2C+M%2E%2C+Schaul%2C+T%2E%2C+Piot%2C+B%2E%2C+Sendonaris%2C+A%2E%2C+Dulac%2DArnold%2C+G%2E%2C+Osband%2C+I%2E%2C+Agapiou%2C+J%2E%2C+et+al%2E+%282017%29%2E+Learning+from+Demonstrations+for+Real+World+Reinforc)
27. Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer. [find](https://scholar.google.com/scholar?q=Hutter%2C+M%2E+%282005%29%2E+Universal+Artificial+Intelligence%3A+Sequential+Decisions+based+on+Algorithmic+Probability%2E+Springer%2E)
28. Islam, R., Henderson, P., Gomrokchi, M., & Precup, D. (2017). Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control. In ICML Workshop on Reproducibility in Machine Learning. [find](https://scholar.google.com/scholar?q=Islam%2C+R%2E%2C+Henderson%2C+P%2E%2C+Gomrokchi%2C+M%2E%2C+%26+Precup%2C+D%2E+%282017%29%2E+Reproducibility+of+Benchmarked+Deep+Reinforcement+Learning+Tasks+for+Continuous+Control%2E+In+ICML+Workshop+on+Reproducibility+in+Machine+Le)
29. Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2017). [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://md.research.elamur.ai/free/en/papers/jaderberg-2016-reinforcement-learning-unsupervised-auxiliary-tasks/). In Proceedings of the International Conference on Learning Representations (ICLR).
30. Jinnai, Y., & Fukunaga, A. (2017). Learning to Prune Dominated Action Sequences in Online Black-Box Planning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 839–845. [find](https://scholar.google.com/scholar?q=Jinnai%2C+Y%2E%2C+%26+Fukunaga%2C+A%2E+%282017%29%2E+Learning+to+Prune+Dominated+Action+Sequences+in+Online+Black%2DBox+Planning%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2C+pp%2E+839%E2%80%93845%2E)
31. Johnson, M., Hofmann, K., Hutton, T., & Bignell, D. (2016). The Malmo Platform for Artificial Intelligence Experimentation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 4246–4247. [find](https://scholar.google.com/scholar?q=Johnson%2C+M%2E%2C+Hofmann%2C+K%2E%2C+Hutton%2C+T%2E%2C+%26+Bignell%2C+D%2E+%282016%29%2E+The+Malmo+Platform+for+Artificial+Intelligence+Experimentation%2E+In+Proceedings+of+the+International+Joint+Conference+on+Artificial+Intellige)
32. Kearns, M. J., Mansour, Y., & Ng, A. Y. (1999). Approximate Planning in Large POMDPs via Reusable Trajectories. In Advances in Neural Information Processing Systems (NIPS), pp. 1001–1007. [find](https://scholar.google.com/scholar?q=Kearns%2C+M%2E+J%2E%2C+Mansour%2C+Y%2E%2C+%26+Ng%2C+A%2E+Y%2E+%281999%29%2E+Approximate+Planning+in+Large+POMDPs+via+Reusable+Trajectories%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pp%2E+1001%E2%80%931007%2E)
33. Kearns, M. J., & Singh, S. P. (2002). Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49 (2-3), 209–232. [find](https://scholar.google.com/scholar?q=Kearns%2C+M%2E+J%2E%2C+%26+Singh%2C+S%2E+P%2E+%282002%29%2E+Near%2DOptimal+Reinforcement+Learning+in+Polynomial+Time%2E+Machine+Learning%2C+49+%282%2D3%29%2C+209%E2%80%93232%2E)
34. Kelly, S., & Heywood, M. I. (2017). Emergent Tangled Graph Representations for Atari Game Playing Agents. In European Conference on Genetic Programming (EuroGP), pp. 64–79. [find](https://scholar.google.com/scholar?q=Kelly%2C+S%2E%2C+%26+Heywood%2C+M%2E+I%2E+%282017%29%2E+Emergent+Tangled+Graph+Representations+for+Atari+Game+Playing+Agents%2E+In+European+Conference+on+Genetic+Programming+%28EuroGP%29%2C+pp%2E+64%E2%80%9379%2E)
35. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). [ImageNet Classification with Deep Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in Neural Information Processing Systems (NIPS), pp. 1106–1114.
36. Levine, J., Congdon, C. B., Ebner, M., Kendall, G., Lucas, S. M., Miikkulainen, R., Schaul, T., & Thompson, T. (2013). General Video Game Playing. In Dagstuhl Follow-Ups. [find](https://scholar.google.com/scholar?q=Levine%2C+J%2E%2C+Congdon%2C+C%2E+B%2E%2C+Ebner%2C+M%2E%2C+Kendall%2C+G%2E%2C+Lucas%2C+S%2E+M%2E%2C+Miikkulainen%2C+R%2E%2C+Schaul%2C+T%2E%2C+%26+Thompson%2C+T%2E+%282013%29%2E+General+Video+Game+Playing%2E+In+Dagstuhl+Follow%2DUps%2E)
37. Liang, Y., Machado, M. C., Talvitie, E., & Bowling, M. H. (2016). State of the Art Control of Atari Games Using Shallow Reinforcement Learning. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems (AAMAS), pp. 485–493. [find](https://scholar.google.com/scholar?q=Liang%2C+Y%2E%2C+Machado%2C+M%2E+C%2E%2C+Talvitie%2C+E%2E%2C+%26+Bowling%2C+M%2E+H%2E+%282016%29%2E+State+of+the+Art+Control+of+Atari+Games+Using+Shallow+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Conference+on+Autono)
38. Lin, L.-J. (1993). Reinforcement Learning for Robots Using Neural Networks. Tech. rep., Carnegie Mellon University, School of Computer Science. [find](https://scholar.google.com/scholar?q=Lin%2C+L%2E%2DJ%2E+%281993%29%2E+Reinforcement+Learning+for+Robots+Using+Neural+Networks%2E+Tech%2E+rep%2E%2C+Carnegie+Mellon+University%2C+School+of+Computer+Science%2E)
39. Lipovetzky, N., Ramirez, M., & Geffner, H. (2015). Classical Planning with Simulators: Results on the Atari Video Games. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 1610–1616. [find](https://scholar.google.com/scholar?q=Lipovetzky%2C+N%2E%2C+Ramirez%2C+M%2E%2C+%26+Geffner%2C+H%2E+%282015%29%2E+Classical+Planning+with+Simulators%3A+Results+on+the+Atari+Video+Games%2E+In+Proceedings+of+the+International+Joint+Conference+on+Artificial+Intelligence)
40. Machado, M. C., Bellemare, M. G., & Bowling, M. (2017). A Laplacian Framework for Option Discovery in Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Machado%2C+M%2E+C%2E%2C+Bellemare%2C+M%2E+G%2E%2C+%26+Bowling%2C+M%2E+%282017%29%2E+A+Laplacian+Framework+for+Option+Discovery+in+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2E)
41. Machado, M. C., Srinivasan, S., & Bowling, M. (2015). Domain-Independent Optimistic Initialization for Reinforcement Learning. In AAAI Workshop on Learning for General Competency in Video Games. [find](https://scholar.google.com/scholar?q=Machado%2C+M%2E+C%2E%2C+Srinivasan%2C+S%2E%2C+%26+Bowling%2C+M%2E+%282015%29%2E+Domain%2DIndependent+Optimistic+Initialization+for+Reinforcement+Learning%2E+In+AAAI+Workshop+on+Learning+for+General+Competency+in+Video+Games%2E)
42. Maei, H. R., & Sutton, R. S. (2010). GQ(λ): A General Gradient Algorithm for Temporal-Difference Prediction Learning with Eligibility Traces. In Proceedings of the Conference on Artificial General Intelligence (AGI), pp. 719–726. [find](https://scholar.google.com/scholar?q=Maei%2C+H%2E+R%2E%2C+%26+Sutton%2C+R%2E+S%2E+%282010%29%2E+GQ%28%CE%BB%29%3A+A+General+Gradient+Algorithm+for+Temporal%2DDifference+Prediction+Learning+with+Eligibility+Traces%2E+In+Proceedings+of+the+Conference+on+Artificial+General+Int)
43. Martin, J., Sasikumar, S. N., Everitt, T., & Hutter, M. (2017). Count-Based Exploration in Feature Space for Reinforcement Learning. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). [find](https://scholar.google.com/scholar?q=Martin%2C+J%2E%2C+Sasikumar%2C+S%2E+N%2E%2C+Everitt%2C+T%2E%2C+%26+Hutter%2C+M%2E+%282017%29%2E+Count%2DBased+Exploration+in+Feature+Space+for+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Joint+Conference+on+Artificial+)
44. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1928–1937. [find](https://scholar.google.com/scholar?q=Mnih%2C+V%2E%2C+Badia%2C+A%2E+P%2E%2C+Mirza%2C+M%2E%2C+Graves%2C+A%2E%2C+Lillicrap%2C+T%2E+P%2E%2C+Harley%2C+T%2E%2C+Silver%2C+D%2E%2C+%26+Kavukcuoglu%2C+K%2E+%282016%29%2E+Asynchronous+Methods+for+Deep+Reinforcement+Learning%2E+In+Proceedings+of+the+Internati)
45. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level Control through Deep… [find](https://scholar.google.com/scholar?q=Mnih%2C+V%2E%2C+Kavukcuoglu%2C+K%2E%2C+Silver%2C+D%2E%2C+Rusu%2C+A%2E+A%2E%2C+Veness%2C+J%2E%2C+Bellemare%2C+M%2E+G%2E%2C+Graves%2C+A%2E%2C+Riedmiller%2C+M%2E%2C+Fidjeland%2C+A%2E+K%2E%2C+Ostrovski%2C+G%2E%2C+Petersen%2C+S%2E%2C+Beattie%2C+C%2E%2C+Sadik%2C+A%2E%2C+Antonoglou%2C+I%2E%2C+Kin)
46. Montfort, N., & Bogost, I. (2009). Racing the Beam: The Atari Video Computer System. MIT Press. [find](https://scholar.google.com/scholar?q=Montfort%2C+N%2E%2C+%26+Bogost%2C+I%2E+%282009%29%2E+Racing+the+Beam%3A+The+Atari+Video+Computer+System%2E+MIT+Press%2E)
47. Munos, R., Stepleton, T., Harutyunyan, A., & Bellemare, M. G. (2016). Safe and Efficient Off-Policy Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Munos%2C+R%2E%2C+Stepleton%2C+T%2E%2C+Harutyunyan%2C+A%2E%2C+%26+Bellemare%2C+M%2E+G%2E+%282016%29%2E+Safe+and+Efficient+Off%2DPolicy+Reinforcement+Learning%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E)
48. Naddaf, Y. (2010). Game-independent AI Agents for Playing Atari 2600 Console Games. Master’s thesis, University of Alberta. [find](https://scholar.google.com/scholar?q=Naddaf%2C+Y%2E+%282010%29%2E+Game%2Dindependent+AI+Agents+for+Playing+Atari+2600+Console+Games%2E+Master%E2%80%99s+thesis%2C+University+of+Alberta%2E)
49. Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. D., Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., & Silver, D. (2015). [Massively Parallel Methods for Deep Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/nair-2015-massively-parallel-methods-deep-reinforcement/). In ICML Deep Learning Workshop.
50. Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015). [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://md.research.elamur.ai/free/en/papers/oh-2015-action-conditional-video-prediction-using/). In Advances in Neural Information Processing Systems (NIPS), pp. 2863–2871.
51. Ontanon, S., Synnaeve, G., Uriarte, A., Richoux, F., Churchill, D., & Preuss, M. (2013). A Survey of Real-Time Strategy Game AI Research and Competition in StarCraft. IEEE Transactions on Computational Intelligence and AI in Games, 5, 293–311. [find](https://scholar.google.com/scholar?q=Ontanon%2C+S%2E%2C+Synnaeve%2C+G%2E%2C+Uriarte%2C+A%2E%2C+Richoux%2C+F%2E%2C+Churchill%2C+D%2E%2C+%26+Preuss%2C+M%2E+%282013%29%2E+A+Survey+of+Real%2DTime+Strategy+Game+AI+Research+and+Competition+in+StarCraft%2E+IEEE+Transactions+on+Computationa)
52. Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016). [Deep Exploration via Bootstrapped DQN](https://md.research.elamur.ai/free/en/papers/osband-2016-deep-exploration-bootstrapped-dqn/). In Advances in Neural Information Processing Systems (NIPS), pp. 4026–4034.
53. Ostrovski, G., Bellemare, M. G., van den Oord, A., & Munos, R. (2017). Count-Based Exploration with Neural Density Models. In Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Ostrovski%2C+G%2E%2C+Bellemare%2C+M%2E+G%2E%2C+van+den+Oord%2C+A%2E%2C+%26+Munos%2C+R%2E+%282017%29%2E+Count%2DBased+Exploration+with+Neural+Density+Models%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2E)
54. Oudeyer, P., Kaplan, F., & Hafner, V. (2007). Intrinsic Motivation Systems for Autonomous Mental Development. IEEE Transactions on Evolutionary Computation, 11 (2), 265– 286. [find](https://scholar.google.com/scholar?q=Oudeyer%2C+P%2E%2C+Kaplan%2C+F%2E%2C+%26+Hafner%2C+V%2E+%282007%29%2E+Intrinsic+Motivation+Systems+for+Autonomous+Mental+Development%2E+IEEE+Transactions+on+Evolutionary+Computation%2C+11+%282%29%2C+265%E2%80%93+286%2E)
55. Parisotto, E., Ba, L. J., & Salakhutdinov, R. (2016). Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Parisotto%2C+E%2E%2C+Ba%2C+L%2E+J%2E%2C+%26+Salakhutdinov%2C+R%2E+%282016%29%2E+Actor%2DMimic%3A+Deep+Multitask+and+Transfer+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29)
56. Ring, M. (1997). CHILD: A First Step Towards Continual Learning. Machine Learning, 28 (1), 77–104. [find](https://scholar.google.com/scholar?q=Ring%2C+M%2E+%281997%29%2E+CHILD%3A+A+First+Step+Towards+Continual+Learning%2E+Machine+Learning%2C+28+%281%29%2C+77%E2%80%93104%2E)
57. Rummery, G. A., & Niranjan, M. (1994). On-line Q-Learning using Connectionist Systems. CUED/F-INFENG/TR 166, Cambridge University Engineering Dept. [find](https://scholar.google.com/scholar?q=Rummery%2C+G%2E+A%2E%2C+%26+Niranjan%2C+M%2E+%281994%29%2E+On%2Dline+Q%2DLearning+using+Connectionist+Systems%2E+CUED%2FF%2DINFENG%2FTR+166%2C+Cambridge+University+Engineering+Dept%2E)
58. Rusu, A. A., Colmenarejo, S. G., Gucehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., & Hadsell, R. (2016). Policy Distillation. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Rusu%2C+A%2E+A%2E%2C+Colmenarejo%2C+S%2E+G%2E%2C+Gucehre%2C+C%2E%2C+Desjardins%2C+G%2E%2C+Kirkpatrick%2C+J%2E%2C+Pascanu%2C+R%2E%2C+Mnih%2C+V%2E%2C+Kavukcuoglu%2C+K%2E%2C+%26+Hadsell%2C+R%2E+%282016%29%2E+Policy+Distillation%2E+In+Proceedings+of+the+International+Co)
59. Schaeffer, J., Burch, N., Bjrnsson, Y., Kishimoto, A., Müller, M., Lake, R., Lu, P., & Sut-phen, S. (2007). Checkers is Solved. Science, 317 (5844), 1518–1522. [find](https://scholar.google.com/scholar?q=Schaeffer%2C+J%2E%2C+Burch%2C+N%2E%2C+Bjrnsson%2C+Y%2E%2C+Kishimoto%2C+A%2E%2C+Mu%CC%88ller%2C+M%2E%2C+Lake%2C+R%2E%2C+Lu%2C+P%2E%2C+%26+Sut%2Dphen%2C+S%2E+%282007%29%2E+Checkers+is+Solved%2E+Science%2C+317+%285844%29%2C+1518%E2%80%931522%2E)
60. Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Schaul%2C+T%2E%2C+Quan%2C+J%2E%2C+Antonoglou%2C+I%2E%2C+%26+Silver%2C+D%2E+%282016%29%2E+Prioritized+Experience+Replay%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
61. Shleyfman, A., Tuisov, A., & Domshlak, C. (2016). Blind Search for Atari-Like Online Planning Revisited. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 3251–3257. [find](https://scholar.google.com/scholar?q=Shleyfman%2C+A%2E%2C+Tuisov%2C+A%2E%2C+%26+Domshlak%2C+C%2E+%282016%29%2E+Blind+Search+for+Atari%2DLike+Online+Planning+Revisited%2E+In+Proceedings+of+the+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%29%2C+pp%2E+32)
62. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). M… [find](https://scholar.google.com/scholar?q=Silver%2C+D%2E%2C+Huang%2C+A%2E%2C+Maddison%2C+C%2E+J%2E%2C+Guez%2C+A%2E%2C+Sifre%2C+L%2E%2C+van+den+Driessche%2C+G%2E%2C+Schrittwieser%2C+J%2E%2C+Antonoglou%2C+I%2E%2C+Panneershelvam%2C+V%2E%2C+Lanctot%2C+M%2E%2C+Dieleman%2C+S%2E%2C+Grewe%2C+D%2E%2C+Nham%2C+J%2E%2C+Kalchbrenner%2C)
63. Singh, S., Barto, A. G., & Chentanez, N. (2004). Intrinsically Motivated Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Singh%2C+S%2E%2C+Barto%2C+A%2E+G%2E%2C+%26+Chentanez%2C+N%2E+%282004%29%2E+Intrinsically+Motivated+Reinforcement+Learning%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E)
64. Stadie, B. C., Levine, S., & Abbeel, P. (2015). Incentivizing Exploration in Reinforcement Learning With Deep Predictive Models. CoRR, abs/1507.00814. [find](https://scholar.google.com/scholar?q=Stadie%2C+B%2E+C%2E%2C+Levine%2C+S%2E%2C+%26+Abbeel%2C+P%2E+%282015%29%2E+Incentivizing+Exploration+in+Reinforcement+Learning+With+Deep+Predictive+Models%2E+CoRR%2C+abs%2F1507%2E00814%2E)
65. Strehl, A. L., & Littman, M. L. (2008). An Analysis of Model-Based Interval Estimation for Markov Decision Processes. Journal of Computer and System Sciences, 74 (8), 1309–1331. [find](https://scholar.google.com/scholar?q=Strehl%2C+A%2E+L%2E%2C+%26+Littman%2C+M%2E+L%2E+%282008%29%2E+An+Analysis+of+Model%2DBased+Interval+Estimation+for+Markov+Decision+Processes%2E+Journal+of+Computer+and+System+Sciences%2C+74+%288%29%2C+1309%E2%80%931331%2E)
66. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press. [find](https://scholar.google.com/scholar?q=Sutton%2C+R%2E+S%2E%2C+%26+Barto%2C+A%2E+G%2E+%281998%29%2E+Reinforcement+Learning%3A+An+Introduction%2E+MIT+Press%2E)
67. Sutton, R., Modayil, J., Delp, M., Degris, T., Pilarski, P., White, A., & Precup, D. (2011). Horde: A Scalable Real-Time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems (AAMAS), p… [find](https://scholar.google.com/scholar?q=Sutton%2C+R%2E%2C+Modayil%2C+J%2E%2C+Delp%2C+M%2E%2C+Degris%2C+T%2E%2C+Pilarski%2C+P%2E%2C+White%2C+A%2E%2C+%26+Precup%2C+D%2E+%282011%29%2E+Horde%3A+A+Scalable+Real%2DTime+Architecture+for+Learning+Knowledge+from+Unsupervised+Sensorimotor+Interaction%2E)
68. Talvitie, E. (2014). Model Regularization for Stable Sample Rollouts. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), pp. 780–789. [find](https://scholar.google.com/scholar?q=Talvitie%2C+E%2E+%282014%29%2E+Model+Regularization+for+Stable+Sample+Rollouts%2E+In+Proceedings+of+the+Conference+on+Uncertainty+in+Artificial+Intelligence+%28UAI%29%2C+pp%2E+780%E2%80%93789%2E)
69. Talvitie, E. (2017). Self-correcting models for model-based reinforcement learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 2597–2603. [find](https://scholar.google.com/scholar?q=Talvitie%2C+E%2E+%282017%29%2E+Self%2Dcorrecting+models+for+model%2Dbased+reinforcement+learning%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2C+pp%2E+2597%E2%80%932603%2E)
70. Taylor, M. E., & Stone, P. (2009). Transfer Learning for Reinforcement Learning Domains: A Survey. Journal of Machine Learning Research, 10, 1633–1685. [find](https://scholar.google.com/scholar?q=Taylor%2C+M%2E+E%2E%2C+%26+Stone%2C+P%2E+%282009%29%2E+Transfer+Learning+for+Reinforcement+Learning+Domains%3A+A+Survey%2E+Journal+of+Machine+Learning+Research%2C+10%2C+1633%E2%80%931685%2E)
71. Thrun, S., & Mitchell, T. M. (1993). [find](https://scholar.google.com/scholar?q=Thrun%2C+S%2E%2C+%26+Mitchell%2C+T%2E+M%2E+%281993%29%2E)
72. Lifelong Robot Learning. In Robotics and Autonomous Systems. van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 2094–2100. [find](https://scholar.google.com/scholar?q=Lifelong+Robot+Learning%2E+In+Robotics+and+Autonomous+Systems%2E+van+Hasselt%2C+H%2E%2C+Guez%2C+A%2E%2C+%26+Silver%2C+D%2E+%282016%29%2E+Deep+Reinforcement+Learning+with+Double+Q%2DLearning%2E+In+Proceedings+of+the+Conference+on+Art)
73. Veness, J., Bellemare, M. G., Hutter, M., Chua, A., & Desjardins, G. (2015). Compress and Control. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 3016–3023. [find](https://scholar.google.com/scholar?q=Veness%2C+J%2E%2C+Bellemare%2C+M%2E+G%2E%2C+Hutter%2C+M%2E%2C+Chua%2C+A%2E%2C+%26+Desjardins%2C+G%2E+%282015%29%2E+Compress+and+Control%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2C+pp%2E+3016%E2%80%933023%2E)
74. Venkatraman, A., Hebert, M., & Bagnell, J. A. (2015). Improving Multi-Step Prediction of Learned Time Series Models. In Proceedings of the Conference on Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=Venkatraman%2C+A%2E%2C+Hebert%2C+M%2E%2C+%26+Bagnell%2C+J%2E+A%2E+%282015%29%2E+Improving+Multi%2DStep+Prediction+of+Learned+Time+Series+Models%2E+In+Proceedings+of+the+Conference+on+Artificial+Intelligence+%28AAAI%29%2E)
75. Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017). FeUdal Networks for Hierarchical Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Vezhnevets%2C+A%2E+S%2E%2C+Osindero%2C+S%2E%2C+Schaul%2C+T%2E%2C+Heess%2C+N%2E%2C+Jaderberg%2C+M%2E%2C+Silver%2C+D%2E%2C+%26+Kavukcuoglu%2C+K%2E+%282017%29%2E+FeUdal+Networks+for+Hierarchical+Reinforcement+Learning%2E+In+Proceedings+of+the+Internationa)
76. Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & De Freitas, N. (2016). Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%2C+Schaul%2C+T%2E%2C+Hessel%2C+M%2E%2C+van+Hasselt%2C+H%2E%2C+Lanctot%2C+M%2E%2C+%26+De+Freitas%2C+N%2E+%282016%29%2E+Dueling+Network+Architectures+for+Deep+Reinforcement+Learning%2E+In+Proceedings+of+the+International+Conference+o)
77. Wilson, S. (1985). Knowledge Growth in an Artificial Animal. In Proceedings of the International Conference on Genetic Algorithms (ICGA). [find](https://scholar.google.com/scholar?q=Wilson%2C+S%2E+%281985%29%2E+Knowledge+Growth+in+an+Artificial+Animal%2E+In+Proceedings+of+the+International+Conference+on+Genetic+Algorithms+%28ICGA%29%2E)
78. Zahavy, T., Ben-Zrihem, N., & Mannor, S. (2016). Graying the Black Box: Understanding DQNs. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1899–1908. [find](https://scholar.google.com/scholar?q=Zahavy%2C+T%2E%2C+Ben%2DZrihem%2C+N%2E%2C+%26+Mannor%2C+S%2E+%282016%29%2E+Graying+the+Black+Box%3A+Understanding+DQNs%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2C+pp%2E+1899%E2%80%931908%2E)
79. O 1 H 1 A 2 O 2 H 2 Figure 9: History tree representation of an environment. [find](https://scholar.google.com/scholar?q=O+1+H+1+A+2+O+2+H+2+Figure+9%3A+History+tree+representation+of+an+environment%2E)

*Updated 2 November 2026*