---
title: "Automatic Goal Generation for Reinforcement Learning Agents"
description: "The generator then proposes new goals with that property. 45 references, cited by 43 papers here."
authors: ["Carlos Florensa", "David Held", "Xinyang Geng", "Pieter Abbeel"]
year: 2017
venue: "arXiv:1705.06366"
source: "https://arxiv.org/abs/1705.06366"
doi: "10.48550/arxiv.1705.06366"
arxiv: "1705.06366"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/florensa-2017-automatic-goal-generation-reinforcement-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/florensa-2017-automatic-goal-generation-reinforcement-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Automatic Goal Generation for Reinforcement Learning Agents

# Automatic Goal Generation for Reinforcement Learning Agents

**Authors:** Carlos Florensa, David Held, [Xinyang Geng](https://md.research.elamur.ai/free/en/authors/xinyang-geng/) and [Pieter Abbeel](https://md.research.elamur.ai/free/en/authors/pieter-abbeel/) · **Year:** 2017 · **Venue:** arXiv:1705.06366 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1705.06366) · [PDF](https://arxiv.org/pdf/1705.06366) · [DOI 10.48550/arxiv.1705.06366](https://doi.org/10.48550/arxiv.1705.06366) · [arXiv 1705.06366](https://arxiv.org/abs/1705.06366) · [OpenAlex](https://openalex.org/W2616430965)

**Citations:** 138 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/florensa-2017-automatic-goal-generation-reinforcement-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard reinforcement learning trains an agent to optimize a single reward, which works poorly when a robot must handle many related tasks, such as reaching arbitrary positions. Florensa, Held, Geng and Abbeel (ICML 2018) aim to maximize the average success rate across all feasible goals. Each goal is a region of the state space, and the agent gets a sparse binary reward only when it enters that region.

Their key idea is the Goal GAN, an adversarially trained generative model. The discriminator learns which goals are of intermediate difficulty for the current policy, meaning neither always nor never reached. The generator then proposes new goals with that property. Training cycles through three steps: label goals, refit the generator, and update the goal-conditioned policy. This produces a curriculum that keeps pace with the agent and needs no prior knowledge of the environment or reward shaping.

Experiments with point-mass and Ant agents show that the method reaches near-full goal coverage more efficiently than uniform goal sampling and SAGG-RIAC. Its advantage grows when feasible goals occupy only a small part of the space.

## Key points
- **Goal labeling:** goals are labeled by their empirical success rate. Those falling between thresholds R_min and R_max count as appropriately difficult. A regularized replay buffer of earlier goals is mixed in, for example 200 goals from the Goal GAN plus 100 from replay per iteration.
- **Multi-path point-mass task:** coverage rises from 0.014 at iteration 1 to 0.53 at iteration 10, 0.78 at iteration 30 and 0.98 at iteration 100. The generator tracks several difficulty "fronts" at once without knowing how many modes exist.
- **N-dimensional Point Mass:**
  - The state space is the hypercube [−5, 5]^N.
  - Feasible states make up 20% of the space in 2D and only about 0.00023:1 at N = 6.
  - Uniform sampling degrades sharply as N grows, while Goal GAN stays robust.
  - Results are measured after 200 outer iterations, averaged over 5 seeds.
- **Proposed future work:** combining the method with HER (hindsight experience replay) and building hierarchical policies on top of the multi-goal policy. Code and videos are released on the project site.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Carlos Florensa, David Held, Xinyang Geng and Pieter Abbeel. 2017. Automatic Goal Generation for Reinforcement Learning Agents. arXiv:1705.06366.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{florensa2017automatic,
  title = {{Automatic Goal Generation for Reinforcement Learning Agents}},
  author = {Carlos Florensa, David Held, Xinyang Geng and Pieter Abbeel},
  year = {2017},
  doi = {10.48550/arxiv.1705.06366},
  eprint = {1705.06366},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.06366}
}
```

- [← previous Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings](https://md.research.elamur.ai/free/en/papers/gonzales-2017-improving-word-sense-disambiguation-neural/)
- [next → Neural Discourse Structure for Text Categorization](https://md.research.elamur.ai/free/en/papers/ji-2017-neural-discourse-structure-text-categorization/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

45 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Achiam, J. and Sastry, S. Surprise-based intrinsic motivation for deep reinforcement learning. In 2016 Deep Reinforcement Learning Workshop at NIPS, 2016. [find](https://scholar.google.com/scholar?q=Achiam%2C+J%2E+and+Sastry%2C+S%2E+Surprise%2Dbased+intrinsic+motivation+for+deep+reinforcement+learning%2E+In+2016+Deep+Reinforcement+Learning+Workshop+at+NIPS%2C+2016%2E)
2. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. [Hindsight experience replay](https://md.research.elamur.ai/free/en/papers/andrychowicz-2017-hindsight-experience-replay/). In Advances in Neural Information Processing Systems, 2017.
3. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein GAN. In Stat, 2017. [find](https://scholar.google.com/scholar?q=Arjovsky%2C+M%2E%2C+Chintala%2C+S%2E%2C+and+Bottou%2C+L%2E+Wasserstein+GAN%2E+In+Stat%2C+2017%2E)
4. Baldassarre, G. and Mirolli, M. Temporal-difference competence-based intrinsic motivation (td-cb-im): A mechanism that uses the td-error as an intrinsic reinforcement for deciding which skill to learn when. Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 255–276, 2012. [find](https://scholar.google.com/scholar?q=Baldassarre%2C+G%2E+and+Mirolli%2C+M%2E+Temporal%2Ddifference+competence%2Dbased+intrinsic+motivation+%28td%2Dcb%2Dim%29%3A+A+mechanism+that+uses+the+td%2Derror+as+an+intrinsic+reinforcement+for+deciding+which+skill+to+learn)
5. Baranes, A. and Oudeyer, P.-Y. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pp. 1766–1773. [find](https://scholar.google.com/scholar?q=Baranes%2C+A%2E+and+Oudeyer%2C+P%2E%2DY%2E+Intrinsically+motivated+goal+exploration+for+active+motor+learning+in+robots%3A+A+case+study%2E+In+Intelligent+Robots+and+Systems+%28IROS%29%2C+2010+IEEE%2FRSJ+International+Confere)
6. Baranes, A. and Oudeyer, P.-Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49–73, 2013a. [find](https://scholar.google.com/scholar?q=Baranes%2C+A%2E+and+Oudeyer%2C+P%2E%2DY%2E+Active+learning+of+inverse+models+with+intrinsically+motivated+goal+exploration+in+robots%2E+Robotics+and+Autonomous+Systems%2C+61%281%29%3A49%E2%80%9373%2C+2013a%2E)
7. Baranes, A. and Oudeyer, P.-Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1), 2013b. [find](https://scholar.google.com/scholar?q=Baranes%2C+A%2E+and+Oudeyer%2C+P%2E%2DY%2E+Active+learning+of+inverse+models+with+intrinsically+motivated+goal+exploration+in+robots%2E+Robotics+and+Autonomous+Systems%2C+61%281%29%2C+2013b%2E)
8. Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Bellemare%2C+M%2E+G%2E%2C+Srinivasan%2C+S%2E%2C+Ostrovski%2C+G%2E%2C+Schaul%2C+T%2E%2C+Saxton%2C+D%2E%2C+and+Munos%2C+R%2E+Unifying+count%2Dbased+exploration+and+intrinsic+motivation%2E+Advances+in+Neural+Information+Processing+Systems%2C+201)
9. Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, 2015. [find](https://scholar.google.com/scholar?q=Bengio%2C+S%2E%2C+Vinyals%2C+O%2E%2C+Jaitly%2C+N%2E%2C+and+Shazeer%2C+N%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2015%2E)
10. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In International Conference on Machine Learning, pp. 41–48. ACM, 2009.
11. Deisenroth, M. P., Neumann, G., Peters, J., et al. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1–2):1–142, 2013. [find](https://scholar.google.com/scholar?q=Deisenroth%2C+M%2E+P%2E%2C+Neumann%2C+G%2E%2C+Peters%2C+J%2E%2C+et+al%2E+A+survey+on+policy+search+for+robotics%2E+Foundations+and+Trends+in+Robotics%2C+2%281%E2%80%932%29%3A1%E2%80%93142%2C+2013%2E)
12. Deisenroth, M. P., Englert, P., Peters, J., and Fox, D. Multi-task policy search for robotics. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pp. 3876–3881. [find](https://scholar.google.com/scholar?q=Deisenroth%2C+M%2E+P%2E%2C+Englert%2C+P%2E%2C+Peters%2C+J%2E%2C+and+Fox%2C+D%2E+Multi%2Dtask+policy+search+for+robotics%2E+In+Robotics+and+Automation+%28ICRA%29%2C+2014+IEEE+International+Conference+on%2C+pp%2E+3876%E2%80%933881%2E)
13. Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Learning, 2016. [find](https://scholar.google.com/scholar?q=Duan%2C+Y%2E%2C+Chen%2C+X%2E%2C+Houthooft%2C+R%2E%2C+Schulman%2C+J%2E%2C+and+Abbeel%2C+P%2E+Benchmarking+deep+reinforcement+learning+for+continuous+control%2E+International+Conference+on+Machine+Learning%2C+2016%2E)
14. Fabisch, A. and Metzen, J. H. Active contextual policy search. Journal of Machine Learning Research, 15(1): 3371–3399, 2014. [find](https://scholar.google.com/scholar?q=Fabisch%2C+A%2E+and+Metzen%2C+J%2E+H%2E+Active+contextual+policy+search%2E+Journal+of+Machine+Learning+Research%2C+15%281%29%3A+3371%E2%80%933399%2C+2014%2E)
15. Florensa, C., Duan, Y., and Abbeel, P. Stochastic neural networks for hierarchical reinforcement learning. In International Conference in Learning Representations, 2017a. [find](https://scholar.google.com/scholar?q=Florensa%2C+C%2E%2C+Duan%2C+Y%2E%2C+and+Abbeel%2C+P%2E+Stochastic+neural+networks+for+hierarchical+reinforcement+learning%2E+In+International+Conference+in+Learning+Representations%2C+2017a%2E)
16. Florensa, C., Held, D., Wulfmeier, M., and Abbeel, P. Reverse curriculum generation for reinforcement learning. Conference on Robot Learning (CoRL), 2017b. [find](https://scholar.google.com/scholar?q=Florensa%2C+C%2E%2C+Held%2C+D%2E%2C+Wulfmeier%2C+M%2E%2C+and+Abbeel%2C+P%2E+Reverse+curriculum+generation+for+reinforcement+learning%2E+Conference+on+Robot+Learning+%28CoRL%29%2C+2017b%2E)
17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. In Advances in neural information processing systems, pp. 2672–2680, 2014. [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2E%2C+Pouget%2DAbadie%2C+J%2E%2C+Mirza%2C+M%2E%2C+Xu%2C+B%2E%2C+Warde%2DFarley%2C+D%2E%2C+Ozair%2C+S%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+Generative+adversarial+networks%2E+In+Advances+in+neural+information+processing+systems%2C+)
18. Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. Automated Curriculum Learning for Neural Networks. arXiv preprint, arXiv:1704.03003, 2017. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Bellemare%2C+M%2E+G%2E%2C+Menick%2C+J%2E%2C+Munos%2C+R%2E%2C+and+Kavukcuoglu%2C+K%2E+Automated+Curriculum+Learning+for+Neural+Networks%2E+arXiv+preprint%2C+arXiv%3A1704%2E03003%2C+2017%2E)
19. Guo, X., Singh, S., Lewis, R., and Lee, H. Deep learning for reward design to improve monte carlo tree search in atari games. Conference on Uncertainty in Artificial Intelligence (UAI), 2016. [find](https://scholar.google.com/scholar?q=Guo%2C+X%2E%2C+Singh%2C+S%2E%2C+Lewis%2C+R%2E%2C+and+Lee%2C+H%2E+Deep+learning+for+reward+design+to+improve+monte+carlo+tree+search+in+atari+games%2E+Conference+on+Uncertainty+in+Artificial+Intelligence+%28UAI%29%2C+2016%2E)
20. Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv: 1610.05182, 2016. — [arXiv:1610.05182](https://arxiv.org/abs/1610.05182) [find](https://scholar.google.com/scholar?q=Heess%2C+N%2E%2C+Wayne%2C+G%2E%2C+Tassa%2C+Y%2E%2C+Lillicrap%2C+T%2E%2C+Riedmiller%2C+M%2E%2C+and+Silver%2C+D%2E+Learning+and+transfer+of+modulated+locomotor+controllers%2E+arXiv+preprint+arXiv%3A+1610%2E05182%2C+2016%2E+%E2%80%94+%5BarXiv%3A1610%2E05182%5D%28ht)
21. Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. Variational information maximizing exploration. Advances in Neural Information Processing Systems, 2016. [find](https://scholar.google.com/scholar?q=Houthooft%2C+R%2E%2C+Chen%2C+X%2E%2C+Duan%2C+Y%2E%2C+Schulman%2C+J%2E%2C+De+Turck%2C+F%2E%2C+and+Abbeel%2C+P%2E+Variational+information+maximizing+exploration%2E+Advances+in+Neural+Information+Processing+Systems%2C+2016%2E)
22. Karpathy, A. and Van De Panne, M. Curriculum learning for motor skills. In Canadian Conference on Artificial Intelligence, pp. 325–330. Springer, 2012. [find](https://scholar.google.com/scholar?q=Karpathy%2C+A%2E+and+Van+De+Panne%2C+M%2E+Curriculum+learning+for+motor+skills%2E+In+Canadian+Conference+on+Artificial+Intelligence%2C+pp%2E+325%E2%80%93330%2E+Springer%2C+2012%2E)
23. Konidaris, G., Kuindersma, S., Grupen, R. A., and Barto, A. G. Autonomous skill acquisition on a mobile manipulator. In AAAI, 2011. [find](https://scholar.google.com/scholar?q=Konidaris%2C+G%2E%2C+Kuindersma%2C+S%2E%2C+Grupen%2C+R%2E+A%2E%2C+and+Barto%2C+A%2E+G%2E+Autonomous+skill+acquisition+on+a+mobile+manipulator%2E+In+AAAI%2C+2011%2E)
24. Kupcsik, A. G., Deisenroth, M. P., Peters, J., Neumann, G., and Others. Data-efficient generalization of robot skills with contextual policy search. In AAAI Conference on Artificial Intelligence, pp. 1401–1407, 2013. [find](https://scholar.google.com/scholar?q=Kupcsik%2C+A%2E+G%2E%2C+Deisenroth%2C+M%2E+P%2E%2C+Peters%2C+J%2E%2C+Neumann%2C+G%2E%2C+and+Others%2E+Data%2Defficient+generalization+of+robot+skills+with+contextual+policy+search%2E+In+AAAI+Conference+on+Artificial+Intelligence%2C+pp%2E+)
25. Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016. [find](https://scholar.google.com/scholar?q=Levine%2C+S%2E%2C+Finn%2C+C%2E%2C+Darrell%2C+T%2E%2C+and+Abbeel%2C+P%2E+End%2Dto%2Dend+training+of+deep+visuomotor+policies%2E+Journal+of+Machine+Learning+Research%2C+17%2839%29%3A1%E2%80%9340%2C+2016%2E)
26. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint, arXiv: 1509.02971, 2015. [find](https://scholar.google.com/scholar?q=Lillicrap%2C+T%2E+P%2E%2C+Hunt%2C+J%2E+J%2E%2C+Pritzel%2C+A%2E%2C+Heess%2C+N%2E%2C+Erez%2C+T%2E%2C+Tassa%2C+Y%2E%2C+Silver%2C+D%2E%2C+and+Wierstra%2C+D%2E+Continuous+control+with+deep+reinforcement+learning%2E+arXiv+preprint%2C+arXiv%3A+1509%2E02971%2C+2015%2E)
27. Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Smolley, S. P. On the effectiveness of least squares generative adversarial networks. arXiv preprint arXiv:1712.06391, 2017. — [arXiv:1712.06391](https://arxiv.org/abs/1712.06391) [find](https://scholar.google.com/scholar?q=Mao%2C+X%2E%2C+Li%2C+Q%2E%2C+Xie%2C+H%2E%2C+Lau%2C+R%2E+Y%2E%2C+Wang%2C+Z%2E%2C+and+Smolley%2C+S%2E+P%2E+On+the+effectiveness+of+least+squares+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1712%2E06391%2C+2017%2E+%E2%80%94+%5BarXiv%3A1712%2E06391%5D%28ht)
28. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529–533, 2015. [find](https://scholar.google.com/scholar?q=Mnih%2C+V%2E%2C+Kavukcuoglu%2C+K%2E%2C+Silver%2C+D%2E%2C+Rusu%2C+A%2E+A%2E%2C+Veness%2C+J%2E%2C+Bellemare%2C+M%2E+G%2E%2C+Graves%2C+A%2E%2C+Riedmiller%2C+M%2E%2C+Fidjeland%2C+A%2E+K%2E%2C+Ostrovski%2C+G%2E%2C+et+al%2E+Human%2Dlevel+control+through+deep+reinforcement+lea)
29. Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., Kavukcuoglu, K., et al. Strategic attentive writer for learning macro-actions. ArXiv preprint, ArXiv:1606.04695, 2016. [find](https://scholar.google.com/scholar?q=Mnih%2C+V%2E%2C+Agapiou%2C+J%2E%2C+Osindero%2C+S%2E%2C+Graves%2C+A%2E%2C+Vinyals%2C+O%2E%2C+Kavukcuoglu%2C+K%2E%2C+et+al%2E+Strategic+attentive+writer+for+learning+macro%2Dactions%2E+ArXiv+preprint%2C+ArXiv%3A1606%2E04695%2C+2016%2E)
30. Ranchod, P., Rosman, B., and Konidaris, G. Nonparametric bayesian reward segmentation for skill discovery using inverse reinforcement learning. In International Conference on Intelligent Robots and Systems, pp. 471–477. IEEE, 2015. [find](https://scholar.google.com/scholar?q=Ranchod%2C+P%2E%2C+Rosman%2C+B%2E%2C+and+Konidaris%2C+G%2E+Nonparametric+bayesian+reward+segmentation+for+skill+discovery+using+inverse+reinforcement+learning%2E+In+International+Conference+on+Intelligent+Robots+and+Sy)
31. Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International Conference on Machine Learning, pp. 1312–1320, 2015. [find](https://scholar.google.com/scholar?q=Schaul%2C+T%2E%2C+Horgan%2C+D%2E%2C+Gregor%2C+K%2E%2C+and+Silver%2C+D%2E+Universal+value+function+approximators%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1312%E2%80%931320%2C+2015%2E)
32. Schmidhuber, J. Curious model-building control systems. In International Joint Conference on Neural Networks, pp. 1458–1463. IEEE, 1991. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E+Curious+model%2Dbuilding+control+systems%2E+In+International+Joint+Conference+on+Neural+Networks%2C+pp%2E+1458%E2%80%931463%2E+IEEE%2C+1991%2E)
33. Schmidhuber, J. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development, 2(3):230–247, 2010. [find](https://scholar.google.com/scholar?q=Schmidhuber%2C+J%2E+Formal+theory+of+creativity%2C+fun%2C+and+intrinsic+motivation+%281990%E2%80%932010%29%2E+IEEE+Transactions+on+Autonomous+Mental+Development%2C+2%283%29%3A230%E2%80%93247%2C+2010%2E)
34. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015a. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+Levine%2C+S%2E%2C+Abbeel%2C+P%2E%2C+Jordan%2C+M%2E%2C+and+Moritz%2C+P%2E+Trust+region+policy+optimization%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+1889%E2%80%931897%2C+2015a%2E)
35. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-Dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representation, 2015b. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+Moritz%2C+P%2E%2C+Levine%2C+S%2E%2C+Jordan%2C+M%2E%2C+and+Abbeel%2C+P%2E+High%2DDimensional+continuous+control+using+generalized+advantage+estimation%2E+In+International+Conference+on+Learning+Representation%2C+201)
36. Sharma, S. and Ravindran, B. Online Multi-Task Learning Using Biased Sampling. arXiv preprint arXiv: 1702.06053, 2017. — [arXiv:1702.06053](https://arxiv.org/abs/1702.06053) [find](https://scholar.google.com/scholar?q=Sharma%2C+S%2E+and+Ravindran%2C+B%2E+Online+Multi%2DTask+Learning+Using+Biased+Sampling%2E+arXiv+preprint+arXiv%3A+1702%2E06053%2C+2017%2E+%E2%80%94+%5BarXiv%3A1702%2E06053%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1702%2E06053%29)
37. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=Silver%2C+D%2E%2C+Huang%2C+A%2E%2C+Maddison%2C+C%2E+J%2E%2C+Guez%2C+A%2E%2C+Sifre%2C+L%2E%2C+Van+Den+Driessche%2C+G%2E%2C+Schrittwieser%2C+J%2E%2C+Antonoglou%2C+I%2E%2C+Panneershelvam%2C+V%2E%2C+Lanctot%2C+M%2E%2C+et+al%2E+Mastering+the+game+of+go+with+deep+neural)
38. Sukhbaatar, S., Kostrikov, I., Szlam, A., and Fergus, R. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. arXiv preprint, arXiv: 1703.05407, 2017. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+S%2E%2C+Kostrikov%2C+I%2E%2C+Szlam%2C+A%2E%2C+and+Fergus%2C+R%2E+Intrinsic+Motivation+and+Automatic+Curricula+via+Asymmetric+Self%2DPlay%2E+arXiv+preprint%2C+arXiv%3A+1703%2E05407%2C+2017%2E)
39. Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value iteration networks. In Advances in Neural Information Processing Systems, pp. 2154–2162, 2016. [find](https://scholar.google.com/scholar?q=Tamar%2C+A%2E%2C+Wu%2C+Y%2E%2C+Thomas%2C+G%2E%2C+Levine%2C+S%2E%2C+and+Abbeel%2C+P%2E+Value+iteration+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2154%E2%80%932162%2C+2016%2E)
40. Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, X., Duan, Y., Schulman, J., Turck, F. D., and Abbeel, P. \#exploration: A study of count-based exploration for deep reinforcement learning. CoRR, abs/1611.04717, 2016. [find](https://scholar.google.com/scholar?q=Tang%2C+H%2E%2C+Houthooft%2C+R%2E%2C+Foote%2C+D%2E%2C+Stooke%2C+A%2E%2C+Chen%2C+X%2E%2C+Duan%2C+Y%2E%2C+Schulman%2C+J%2E%2C+Turck%2C+F%2E+D%2E%2C+and+Abbeel%2C+P%2E+%5C%23exploration%3A+A+study+of+count%2Dbased+exploration+for+deep+reinforcement+learning%2E+CoRR%2C+)
41. Tang, Y. and Salakhutdinov, R. Learning Stochastic Feedforward Neural Networks. Advances in Neural Information Processing Systems, 2:530–538, 2013. [find](https://scholar.google.com/scholar?q=Tang%2C+Y%2E+and+Salakhutdinov%2C+R%2E+Learning+Stochastic+Feedforward+Neural+Networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+2%3A530%E2%80%93538%2C+2013%2E)
42. Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012. [find](https://scholar.google.com/scholar?q=Todorov%2C+E%2E%2C+Erez%2C+T%2E%2C+and+Tassa%2C+Y%2E+Mujoco%3A+A+physics+engine+for+model%2Dbased+control%2E+In+IEEE%2FRSJ+International+Conference+on+Intelligent+Robots+and+Systems%2C+2012%2E)
43. Vigorito, C. M. and Barto, A. G. Intrinsically motivated hierarchical skill learning in structured environments. IEEE Transactions on Autonomous Mental Development, 2(2): 132–143, 2010. [find](https://scholar.google.com/scholar?q=Vigorito%2C+C%2E+M%2E+and+Barto%2C+A%2E+G%2E+Intrinsically+motivated+hierarchical+skill+learning+in+structured+environments%2E+IEEE+Transactions+on+Autonomous+Mental+Development%2C+2%282%29%3A+132%E2%80%93143%2C+2010%2E)
44. Zaremba, W. and Sutskever, I. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+W%2E+and+Sutskever%2C+I%2E+Learning+to+execute%2E+arXiv+preprint+arXiv%3A1410%2E4615%2C+2014%2E)
45. Zheng, S., Yue, Y., and Hobbs, J. Generating long-term trajectories using deep hierarchical networks. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29. 2016. [find](https://scholar.google.com/scholar?q=Zheng%2C+S%2E%2C+Yue%2C+Y%2E%2C+and+Hobbs%2C+J%2E+Generating+long%2Dterm+trajectories+using+deep+hierarchical+networks%2E+In+Lee%2C+D%2E+D%2E%2C+Sugiyama%2C+M%2E%2C+Luxburg%2C+U%2E+V%2E%2C+Guyon%2C+I%2E%2C+and+Garnett%2C+R%2E+%28eds%2E%29%2C+Advances+in+Neural)

*Updated 2 November 2026*