---
title: "Action-Conditional Video Prediction using Deep Networks in Atari"
description: "The authors propose two encoder–transformation–decoder networks. 37 references, cited by 48 papers here."
authors: ["Junhyuk Oh", "Xiaoxiao Guo", "Honglak Lee", "Richard Lewis", "Satinder Singh"]
year: 2015
venue: "arXiv:1507.08750"
source: "https://arxiv.org/abs/1507.08750"
doi: "10.48550/arxiv.1507.08750"
arxiv: "1507.08750"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/oh-2015-action-conditional-video-prediction-using/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/oh-2015-action-conditional-video-prediction-using/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Action-Conditional Video Prediction using Deep Networks in Atari Games

# Action-Conditional Video Prediction using Deep Networks in Atari Games

**Authors:** Junhyuk Oh, [Xiaoxiao Guo](https://md.research.elamur.ai/free/en/authors/xiaoxiao-guo/), [Honglak Lee](https://md.research.elamur.ai/free/en/authors/honglak-lee/), Richard Lewis and Satinder Singh · **Year:** 2015 · **Venue:** arXiv:1507.08750 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1507.08750) · [PDF](https://arxiv.org/pdf/1507.08750) · [DOI 10.48550/arxiv.1507.08750](https://doi.org/10.48550/arxiv.1507.08750) · [arXiv 1507.08750](https://arxiv.org/abs/1507.08750) · [OpenAlex](https://openalex.org/W2118688707)

**Citations:** 419 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/oh-2015-action-conditional-video-prediction-using/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper studies whether a deep network can forecast future video frames when those frames depend on the agent's chosen actions as well as on past frames. This amounts to learning an environment model for vision-based reinforcement learning. The experiments are run on Atari titles drawn from the ALE (Arcade Learning Environment) benchmark suite. Their screens are high-dimensional, contain many interacting objects, and are partially observable. The authors propose two encoder–transformation–decoder networks. One uses feedforward convolutional encoding of a stacked frame history. The other uses recurrent encoding, with an LSTM placed on top of CNN features. In both, a factored multiplicative layer combines the encoded features with the action, and deconvolution layers reconstruct the next frame. Training uses a multi-step squared-error loss with a curriculum that gradually lengthens the prediction horizon. In several games, the models produce realistic frames for roughly 100 steps, and these frames remain useful for control with DQN. The learned representations also separate agent-controlled objects from the rest of the scene. The authors present this as the first long-horizon, action-conditioned prediction of high-dimensional video, which makes it a step toward model-based RL from pixels.

## Key points
- **Architecture.** There are two variants: feedforward encoding (early fusion of frames) and recurrent encoding (LSTM without peepholes). Both use a factorized 3-way tensor to condition on actions and deconvolution to decode.
- **Training.** A curriculum moves from short-term to longer-term K-step prediction and was needed for stable training. Optimization uses SGD with BPTT.
- **Replacing emulator frames.** When predicted frames are fed to a pretrained DQN, the score stays well above random play in 3 of 5 games, even with 100-step predictions.
- **Informed exploration.** An ε-greedy variant uses the model to choose the action whose predicted frame is least visited, measured with a Gaussian-kernel visit count over recent frames. This improved DQN in 3 of 5 games, with the largest gain in QBert. It used a feedforward model on 84×84 grayscale input.
- **Representation analysis (Seaquest).**
  - Action factors show positive similarity for shared movement directions and negative similarity for opposing directions.
  - The high-variance factors (about 40%) move only the controlled object.
  - The low-variance factors move the background and other objects.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 86 works · 15,493 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis and Satinder Singh. 2015. Action-Conditional Video Prediction using Deep Networks in Atari Games. arXiv:1507.08750.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{oh2015action,
  title = {{Action-Conditional Video Prediction using Deep Networks in Atari Games}},
  author = {Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis and Satinder Singh},
  year = {2015},
  doi = {10.48550/arxiv.1507.08750},
  eprint = {1507.08750},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1507.08750}
}
```

- [← previous ASTD: Arabic Sentiment Tweets Dataset](https://md.research.elamur.ai/free/en/papers/nabil-2015-astd-arabic-sentiment-tweets-dataset/)
- [next → End-to-end learning of semantic role labeling using recurrent neural networks](https://md.research.elamur.ai/free/en/papers/zhou-2015-end-end-learning-semantic-role/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

37 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. [The Arcade Learning Environment: An evaluation platform for general agents](https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/). Journal of Artificial Intelligence Research, 47:253–279, 2013.
2. M. G. Bellemare, J. Veness, and M. Bowling. Investigating contingency awareness using Atari 2600 games. In AAAI, 2012. [find](https://scholar.google.com/scholar?q=M%2E+G%2E+Bellemare%2C+J%2E+Veness%2C+and+M%2E+Bowling%2E+Investigating+contingency+awareness+using+Atari+2600+games%2E+In+AAAI%2C+2012%2E)
3. M. G. Bellemare, J. Veness, and M. Bowling. Bayesian learning of recursively factored environments. In ICML, 2013. [find](https://scholar.google.com/scholar?q=M%2E+G%2E+Bellemare%2C+J%2E+Veness%2C+and+M%2E+Bowling%2E+Bayesian+learning+of+recursively+factored+environments%2E+In+ICML%2C+2013%2E)
4. M. G. Bellemare, J. Veness, and E. Talvitie. Skip context tree switching. In ICML, 2014. [find](https://scholar.google.com/scholar?q=M%2E+G%2E+Bellemare%2C+J%2E+Veness%2C+and+E%2E+Talvitie%2E+Skip+context+tree+switching%2E+In+ICML%2C+2014%2E)
5. Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1–127, 2009. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2E+Learning+deep+architectures+for+AI%2E+Foundations+and+Trends+in+Machine+Learning%2C+2%281%29%3A1%E2%80%93127%2C+2009%2E)
6. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In ICML, 2009.
7. D. Ciresan, U. Meier, and J. Schmidhuber. [Multi-column deep neural networks for image classification](https://md.research.elamur.ai/free/en/papers/cire-csan-2012-multi-column-deep-neural-networks/). In CVPR, 2012.
8. A. Dosovitskiy, J. T. Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=A%2E+Dosovitskiy%2C+J%2E+T%2E+Springenberg%2C+and+T%2E+Brox%2E+Learning+to+generate+chairs+with+convolutional+neural+networks%2E+In+CVPR%2C+2015%2E)
9. R. Girshick, J. Donahue, T. Darrell, and J. Malik. [Rich feature hierarchies for accurate object detection and semantic segmentation](https://md.research.elamur.ai/free/en/papers/girshick-2013-rich-feature-hierarchies-accurate-object/). In CVPR, 2014.
10. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2C+2013%2E)
11. X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang. Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In NIPS, 2014. [find](https://scholar.google.com/scholar?q=X%2E+Guo%2C+S%2E+Singh%2C+H%2E+Lee%2C+R%2E+L%2E+Lewis%2C+and+X%2E+Wang%2E+Deep+learning+for+real%2Dtime+Atari+game+play+using+offline+Monte%2DCarlo+tree+search+planning%2E+In+NIPS%2C+2014%2E)
12. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
13. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. [Caffe: Convolutional architecture for fast feature embedding](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/). In ACM Multimedia, 2014.
14. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014. [find](https://scholar.google.com/scholar?q=A%2E+Karpathy%2C+G%2E+Toderici%2C+S%2E+Shetty%2C+T%2E+Leung%2C+R%2E+Sukthankar%2C+and+L%2E+Fei%2DFei%2E+Large%2Dscale+video+classification+with+convolutional+neural+networks%2E+In+CVPR%2C+2014%2E)
15. L. Kocsis and C. Szepesvári. Bandit based Monte-Carlo planning. In ECML. 2006. [find](https://scholar.google.com/scholar?q=L%2E+Kocsis+and+C%2E+Szepesva%CC%81ri%2E+Bandit+based+Monte%2DCarlo+planning%2E+In+ECML%2E+2006%2E)
16. A. Krizhevsky, I. Sutskever, and G. E. Hinton. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, 2012.
17. I. Lenz, R. Knepper, and A. Saxena. DeepMPC: Learning deep latent features for model predictive control. In RSS, 2015. [find](https://scholar.google.com/scholar?q=I%2E+Lenz%2C+R%2E+Knepper%2C+and+A%2E+Saxena%2E+DeepMPC%3A+Learning+deep+latent+features+for+model+predictive+control%2E+In+RSS%2C+2015%2E)
18. R. Memisevic. Learning to relate images. IEEE TPAMI, 35(8):1829–1846, 2013. [find](https://scholar.google.com/scholar?q=R%2E+Memisevic%2E+Learning+to+relate+images%2E+IEEE+TPAMI%2C+35%288%29%3A1829%E2%80%931846%2C+2013%2E)
19. V. Michalski, R. Memisevic, and K. Konda. Modeling deep temporal dependencies with recurrent grammar cells. In NIPS, 2014. [find](https://scholar.google.com/scholar?q=V%2E+Michalski%2C+R%2E+Memisevic%2C+and+K%2E+Konda%2E+Modeling+deep+temporal+dependencies+with+recurrent+grammar+cells%2E+In+NIPS%2C+2014%2E)
20. R. Mittelman, B. Kuipers, S. Savarese, and H. Lee. Structured recurrent temporal restricted Boltzmann machines. In ICML, 2014. [find](https://scholar.google.com/scholar?q=R%2E+Mittelman%2C+B%2E+Kuipers%2C+S%2E+Savarese%2C+and+H%2E+Lee%2E+Structured+recurrent+temporal+restricted+Boltzmann+machines%2E+In+ICML%2C+2014%2E)
21. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+Graves%2C+I%2E+Antonoglou%2C+D%2E+Wierstra%2C+and+M%2E+Riedmiller%2E+Playing+Atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312%2E5602%2C+2013%2E)
22. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+A%2E+Rusu%2C+J%2E+Veness%2C+M%2E+G%2E+Bellemare%2C+A%2E+Graves%2C+M%2E+Riedmiller%2C+A%2E+K%2E+Fidjeland%2C+G%2E+Ostrovski%2C+et+al%2E+Human%2Dlevel+control+through+deep+reinforcement+learning%2E+Nat)
23. V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010. [find](https://scholar.google.com/scholar?q=V%2E+Nair+and+G%2E+E%2E+Hinton%2E+Rectified+linear+units+improve+restricted+Boltzmann+machines%2E+In+ICML%2C+2010%2E)
24. S. Reed, K. Sohn, Y. Zhang, and H. Lee. Learning to disentangle factors of variation with manifold interaction. In ICML, 2014. [find](https://scholar.google.com/scholar?q=S%2E+Reed%2C+K%2E+Sohn%2C+Y%2E+Zhang%2C+and+H%2E+Lee%2E+Learning+to+disentangle+factors+of+variation+with+manifold+interaction%2E+In+ICML%2C+2014%2E)
25. S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza. Disentangling factors of variation for facial expression recognition. In ECCV. 2012. [find](https://scholar.google.com/scholar?q=S%2E+Rifai%2C+Y%2E+Bengio%2C+A%2E+Courville%2C+P%2E+Vincent%2C+and+M%2E+Mirza%2E+Disentangling+factors+of+variation+for+facial+expression+recognition%2E+In+ECCV%2E+2012%2E)
26. J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+Deep+learning+in+neural+networks%3A+An+overview%2E+Neural+Networks%2C+61%3A85%E2%80%93117%2C+2015%2E)
27. J. Schmidhuber and R. Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2:125–134, 1991. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber+and+R%2E+Huber%2E+Learning+to+generate+artificial+fovea+trajectories+for+target+detection%2E+International+Journal+of+Neural+Systems%2C+2%3A125%E2%80%93134%2C+1991%2E)
28. N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using LSTMs. In ICML, 2015. [find](https://scholar.google.com/scholar?q=N%2E+Srivastava%2C+E%2E+Mansimov%2C+and+R%2E+Salakhutdinov%2E+Unsupervised+learning+of+video+representations+using+LSTMs%2E+In+ICML%2C+2015%2E)
29. I. Sutskever, G. E. Hinton, and G. W. Taylor. The recurrent temporal restricted Boltzmann machine. In NIPS, 2009. [find](https://scholar.google.com/scholar?q=I%2E+Sutskever%2C+G%2E+E%2E+Hinton%2C+and+G%2E+W%2E+Taylor%2E+The+recurrent+temporal+restricted+Boltzmann+machine%2E+In+NIPS%2C+2009%2E)
30. I. Sutskever, J. Martens, and G. E. Hinton. Generating text with recurrent neural networks. In ICML, 2011. [find](https://scholar.google.com/scholar?q=I%2E+Sutskever%2C+J%2E+Martens%2C+and+G%2E+E%2E+Hinton%2E+Generating+text+with+recurrent+neural+networks%2E+In+ICML%2C+2011%2E)
31. I. Sutskever, O. Vinyals, and Q. Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In NIPS, 2014.
32. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+W%2E+Liu%2C+Y%2E+Jia%2C+P%2E+Sermanet%2C+S%2E+Reed%2C+D%2E+Anguelov%2C+D%2E+Erhan%2C+V%2E+Vanhoucke%2C+and+A%2E+Rabinovich%2E+Going+deeper+with+convolutions%2E+arXiv+preprint+arXiv%3A1409%2E4842%2C+2014%2E)
33. G. W. Taylor and G. E. Hinton. Factored conditional restricted Boltzmann machines for modeling motion style. In ICML, 2009. [find](https://scholar.google.com/scholar?q=G%2E+W%2E+Taylor+and+G%2E+E%2E+Hinton%2E+Factored+conditional+restricted+Boltzmann+machines+for+modeling+motion+style%2E+In+ICML%2C+2009%2E)
34. T. Tieleman and G. Hinton. Lecture 6.5 - RMSProp: Divde the gradient by a running average of its recent magnitude. Coursera, 2012. [find](https://scholar.google.com/scholar?q=T%2E+Tieleman+and+G%2E+Hinton%2E+Lecture+6%2E5+%2D+RMSProp%3A+Divde+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+Coursera%2C+2012%2E)
35. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=D%2E+Tran%2C+L%2E+Bourdev%2C+R%2E+Fergus%2C+L%2E+Torresani%2C+and+M%2E+Paluri%2E+Learning+spatiotemporal+features+with+3D+convolutional+networks%2E+In+ICCV%2C+2015%2E)
36. C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992. [find](https://scholar.google.com/scholar?q=C%2E+J%2E+Watkins+and+P%2E+Dayan%2E+Q%2Dlearning%2E+Machine+learning%2C+8%283%2D4%29%3A279%E2%80%93292%2C+1992%2E)
37. J. Yang, S. Reed, M.-H. Yang, and H. Lee. Weakly-supervised disentangling with recurrent transformations for 3D view synthesis. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=J%2E+Yang%2C+S%2E+Reed%2C+M%2E%2DH%2E+Yang%2C+and+H%2E+Lee%2E+Weakly%2Dsupervised+disentangling+with+recurrent+transformations+for+3D+view+synthesis%2E+In+NIPS%2C+2015%2E)

*Updated 2 November 2026*