---
title: "Gymnasium — paper by Towers et al. (2024)"
description: "This NeurIPS 2025 paper presents Gymnasium, the Farama Foundation's maintained successor. 57 references, cited by 49 papers here."
authors: ["Mark Towers", "Ariel Kwiatkowski", "Jordan Terry", "John U. Balis", "Gianluca De Cola", "Tristan Deleu", "Manuel Goulao", "Andreas Kallinteris", "Markus Krimmel", "Arjun KG", "Rodrigo Perez-Vicente", "Andrea Pierre", "Sander Schulhoff", "Jun Jet Tai", "Hannah Tan", "Omar G. Younis"]
year: 2024
venue: "arXiv:2407.17032"
source: "https://arxiv.org/abs/2407.17032"
doi: "10.48550/arxiv.2407.17032"
arxiv: "2407.17032"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/towers-2024-gymnasium-standard-interface-reinforcement-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/towers-2024-gymnasium-standard-interface-reinforcement-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Gymnasium: A Standard Interface for Reinforcement Learning Environments

# Gymnasium: A Standard Interface for Reinforcement Learning Environments

**Authors:** Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, [Markus Krimmel](https://md.research.elamur.ai/free/en/authors/markus-krimmel/), Arjun KG, Rodrigo Perez-Vicente, Andrea Pierre, Sander Schulhoff, Jun Jet Tai, Hannah Tan and Omar G. Younis · **Year:** 2024 · **Venue:** arXiv:2407.17032 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.17032) · [PDF](https://arxiv.org/pdf/2407.17032) · [DOI 10.48550/arxiv.2407.17032](https://doi.org/10.48550/arxiv.2407.17032) · [arXiv 2407.17032](https://arxiv.org/abs/2407.17032) · [OpenAlex](https://openalex.org/W6891880838)

**Citations:** 64 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/towers-2024-gymnasium-standard-interface-reinforcement-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Progress in reinforcement learning has been slowed by inconsistent environment and algorithm implementations, which makes results hard to compare and reuse. OpenAI Gym once filled the role of a common interface, but OpenAI stopped maintaining it and released no updates after October 2022. This NeurIPS 2025 paper presents Gymnasium, the Farama Foundation's maintained successor. Its core abstractions are:
- **Env**, a loose analogue of a POMDP driven by `reset` and `step`.
- **VectorEnv**, which runs many sub-environments in parallel and batches their outputs.
- **Spaces**, which describe observation and action structure.
- **Registry**, a versioned catalogue that makes environment creation reproducible.

The library also ships wrappers and built-in environments ranging from toy MDPs to robotic simulators. Additions beyond Gym include a functional environment API suited to search methods and hardware acceleration, broader vectorization support with configurable autoreset, and richer spaces for multi-modal agents. Gymnasium is widely adopted and integrates with major training libraries. A stable shared interface cuts engineering effort and improves reproducibility. Its limits are that it is written in Python and does not train agents itself.

## Key points
- **Adoption:** the authors report more than 18 million downloads up to May 2025, over a million in April 2025 alone, roughly 500 GitHub issues, 800 pull requests, and over 40 contributors.
- **Compatible ecosystem:** Stable Baselines3, CleanRL, Tianshou, Ray RLlib, and Dopamine on the training side; Minari, ALE, Metaworld, and Gymnasium Robotics as related tooling.
- **Autoreset modes:** vector environments support three. Next-step is the default and fastest; same-step was Gym's only mode; disabled requires a masked explicit reset.
- **New spaces:** Text, Sequence, Graph, and OneOf (disjoint union) are unique to Gymnasium. They sit alongside Box, Discrete, MultiBinary, MultiDiscrete, Dict, and Tuple.
- **Custom vectorization:** a custom vectorized CartPole built on tensor operations runs roughly ten times more steps per second than the Sync or Async vectorizers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 65 works · 14,425 citations
- [University of Southampton](https://md.research.elamur.ai/free/en/universities/university-of-southampton/) — University, Southampton, United Kingdom · 13 works · 2,608 citations

Also on the byline: [Technical University of Crete](https://ror.org/03f8bz564).

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierre, Sander Schulhoff, Jun Jet Tai, Hannah Tan and Omar G. Younis. 2024. Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv:2407.17032.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{towers2024gymnasium,
  title = {{Gymnasium: A Standard Interface for Reinforcement Learning Environments}},
  author = {Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca {De Cola}, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierre, Sander Schulhoff, Jun Jet Tai, Hannah Tan and Omar G. Younis},
  year = {2024},
  doi = {10.48550/arxiv.2407.17032},
  eprint = {2407.17032},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.17032}
}
```

- [← previous ExpertQA: Expert-Curated Questions and Attributed Answers](https://md.research.elamur.ai/free/en/papers/malaviya-2024-expertqa-expert-curated-questions-attributed/)
- [next → AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://md.research.elamur.ai/free/en/papers/yang-2024-air-bench-benchmarking-large-audio/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

57 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lucas N. Alegre. SUMO-RL. [https://github.com/LucasAlegre/sumo-rl](https://github.com/LucasAlegre/sumo-rl), 2019. [↗](https://github.com/LucasAlegre/sumo-rl)
2. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. [Hindsight experience replay](https://md.research.elamur.ai/free/en/papers/andrychowicz-2017-hindsight-experience-replay/), 2018. URL https://arxiv.org/abs/1707.01495. [↗](https://arxiv.org/abs/1707.01495)
3. Charles Beattie, Thomas Köppe, Edgar A. Duéñez-Guzmán, and Joel Z. Leibo. Deepmind lab2d, 2020. URL https://arxiv.org/abs/2011.07027. — [arXiv:2011.07027](https://arxiv.org/abs/2011.07027) [↗](https://arxiv.org/abs/2011.07027)
4. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. [The arcade learning environment: An evaluation platform for general agents](https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/). Journal of Artificial Intelligence Research, 47:253–279, jun 2013.
5. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. [find](https://scholar.google.com/scholar?q=Christopher+Berner%2C+Greg+Brockman%2C+Brooke+Chan%2C+Vicki+Cheung%2C+Przemys%C5%82aw+Debiak%2C+Christy+Dennison%2C+David+Farhi%2C+Quirin+Fischer%2C+Shariq+Hashme%2C+Chris+Hesse%2C+et+al%2E+Dota+2+with+large+scale+deep+reinforc)
6. Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A.… [find](https://scholar.google.com/scholar?q=Cl%C3%A9ment+Bonnet%2C+Daniel+Luo%2C+Donal+Byrne%2C+Shikha+Surana%2C+Sasha+Abramowitz%2C+Paul+Duckworth%2C+Vincent+Coyette%2C+Laurence+I%2E+Midgley%2C+Elshadai+Tegegn%2C+Tristan+Kalloniatis%2C+Omayma+Mahjoub%2C+Matthew+Macfarlane)
7. Felix Book, Arne Traue, Maximilian Schenke, Barnabas Haucke-Korber, and Oliver Wallscheid. Gym-electric-motor (gem) control: An automated open-source controller design suite for drives. In 2023 IEEE International Electric Machines & Drives Conference (IEMDC), pages 1–7, 2023. doi: 10.1109/IEMDC55163… — [doi:10.1109/iemdc55163.2023.10239044](https://doi.org/10.1109/iemdc55163.2023.10239044) [find](https://scholar.google.com/scholar?q=Felix+Book%2C+Arne+Traue%2C+Maximilian+Schenke%2C+Barnabas+Haucke%2DKorber%2C+and+Oliver+Wallscheid%2E+Gym%2Delectric%2Dmotor+%28gem%29+control%3A+An+automated+open%2Dsource+controller+design+suite+for+drives%2E+In+2023+IEEE+I)
8. Yann Bouteiller, Edouard GEZE, GobeX, Stefan Kuhn, and pius. trackmania-rl/tmrl: Release 0.7.1, May 2025. URL https://doi.org/10.5281/zenodo.15344778. — [doi:10.5281/zenodo.15344778](https://doi.org/10.5281/zenodo.15344778) [↗](https://doi.org/10.5281/zenodo.15344778)
9. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/google/jax](http://github.com/google/jax). [↗](http://github.com/google/jax)
10. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. [find](https://scholar.google.com/scholar?q=Greg+Brockman%2C+Vicki+Cheung%2C+Ludwig+Pettersson%2C+Jonas+Schneider%2C+John+Schulman%2C+Jie+Tang%2C+and+Wojciech+Zaremba%2E+Openai+gym%2E+arXiv+preprint+arXiv%3A1606%2E01540%2C+2016%2E)
11. Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http://arxiv.org/abs/1812.06110. — [arXiv:1812.06110](https://arxiv.org/abs/1812.06110) [↗](http://arxiv.org/abs/1812.06110)
12. Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Process… [find](https://scholar.google.com/scholar?q=Maxime+Chevalier%2DBoisvert%2C+Bolun+Dai%2C+Mark+Towers%2C+Rodrigo+Perez%2DVicente%2C+Lucas+Willems%2C+Salem+Lahlou%2C+Suman+Pal%2C+Pablo+Samuel+Castro%2C+and+Jordan+Terry%2E+Minigrid+%26+miniworld%3A+Modular+%26+customizable+re)
13. Kartik Choudhary, Dhawal Gupta, and Philip S. Thomas. ICU-Sepsis: A benchmark MDP built from real medical data. In Reinforcement Learning Conference, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2406.05646. — [arXiv:2406.05646](https://arxiv.org/abs/2406.05646) [↗](https://arxiv.org/)
14. Steven Dalton, Iuri Frosio, and Michael Garland. Accelerating reinforcement learning through gpu atari emulation, 2020. URL https://arxiv.org/abs/1907.08467. [↗](https://arxiv.org/abs/1907.08467)
15. Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2023. URL [http://github.com/Farama-Foundation/](http://github.com/Farama-Foundation/) Gymnasium-Robotics. [↗](http://github.com/Farama-Foundation/)
16. Gustavo de Moura Souza and Claudio Fabiano Motta Toledo. Monte carlo continuous action space solver for chance constrained markov decision processes. Master’s thesis, Universidade de São Paulo, 2024. [find](https://scholar.google.com/scholar?q=Gustavo+de+Moura+Souza+and+Claudio+Fabiano+Motta+Toledo%2E+Monte+carlo+continuous+action+space+solver+for+chance+constrained+markov+decision+processes%2E+Master%E2%80%99s+thesis%2C+Universidade+de+S%C3%A3o+Paulo%2C+2024%2E)
17. Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting. Ocatari: Object-centric atari 2600 reinforcement learning environments, 2024. URL https://arxiv.org/abs/2306.08649. — [arXiv:2306.08649](https://arxiv.org/abs/2306.08649) [↗](https://arxiv.org/abs/2306.08649)
18. Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. [https://github.com/openai/baselines](https://github.com/openai/baselines), 2017. [↗](https://github.com/openai/baselines)
19. Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 20… [find](https://scholar.google.com/scholar?q=Florian+Felten%2C+Lucas+N%2E+Alegre%2C+Ann+Now%C3%A9%2C+Ana+L%2E+C%2E+Bazzan%2C+El+Ghazali+Talbi%2C+Gr%C3%A9goire+Danoy%2C+and+Bruno+C%2E+da+Silva%2E+A+toolkit+for+reliable+benchmarking+and+research+in+multi%2Dobjective+reinforcement+)
20. C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL [http://github.com/google/brax](http://github.com/google/brax). [↗](http://github.com/google/brax)
21. Xudong Gong, Dawei Feng, kele Xu, Weijia Wang, Zhangjun Sun, Xing Zhou, Bo Ding, and Huaimin Wang. Vvc-gym: A fixed-wing uav reinforcement learning environment for multi-goal long-horizon problems. In International Conference on Learning Representations, 2025. [find](https://scholar.google.com/scholar?q=Xudong+Gong%2C+Dawei+Feng%2C+kele+Xu%2C+Weijia+Wang%2C+Zhangjun+Sun%2C+Xing+Zhou%2C+Bo+Ding%2C+and+Huaimin+Wang%2E+Vvc%2Dgym%3A+A+fixed%2Dwing+uav+reinforcement+learning+environment+for+multi%2Dgoal+long%2Dhorizon+problems%2E+In)
22. DJ Groot, G Leto, A Vlaskin, A Moec, and J Ellerbroek. Bluesky-gym: Reinforcement learning environments for air traffic applications, 2024. [find](https://scholar.google.com/scholar?q=DJ+Groot%2C+G+Leto%2C+A+Vlaskin%2C+A+Moec%2C+and+J+Ellerbroek%2E+Bluesky%2Dgym%3A+Reinforcement+learning+environments+for+air+traffic+applications%2C+2024%2E)
23. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:1801.01290 [cs, stat], January 2018. URL http://arxiv.org/abs/1801.01290. arXiv: 1801.01290 version: 1. [↗](http://arxiv.org/abs/1801.01290)
24. Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Se… — [arXiv:2006.00979](https://arxiv.org/abs/2006.00979) [find](https://scholar.google.com/scholar?q=Matthew+W%2E+Hoffman%2C+Bobak+Shahriari%2C+John+Aslanides%2C+Gabriel+Barth%2DMaron%2C+Nikola+Momchev%2C+Danila+Sinopalnikov%2C+Piotr+Stan%CC%81czyk%2C+Sabela+Ramos%2C+Anton+Raichuk%2C+Damien+Vincent%2C+L%C3%A9onard+Hussenot%2C+Robert+Da)
25. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. URL [http://jmlr.org/papers/…](http://jmlr.org/papers/…) [↗](http://jmlr.org/papers/…)
26. Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichuk, Mohamad H. Danesh, Cyril Roumégous, Jiayi Weng, Chufan Chen, Md Masudur Rahman, João G. M. Araújo, Guorui Quan, Daniel Tan, Timo Klein… — [arXiv:2402.03046](https://arxiv.org/abs/2402.03046) [find](https://scholar.google.com/scholar?q=Shengyi+Huang%2C+Quentin+Gallou%C3%A9dec%2C+Florian+Felten%2C+Antonin+Raffin%2C+Rousslan+Fernand+Julien+Dossa%2C+Yanxiao+Zhao%2C+Ryan+Sullivan%2C+Viktor+Makoviychuk%2C+Denys+Makoviichuk%2C+Mohamad+H%2E+Danesh%2C+Cyril+Roum%C3%A9gous)
27. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99–134, May 1998. ISSN 0004-3702. doi: 10.1016/S0004-3702(98)00023-X. [find](https://scholar.google.com/scholar?q=Leslie+Pack+Kaelbling%2C+Michael+L%2E+Littman%2C+and+Anthony+R%2E+Cassandra%2E+Planning+and+acting+in+partially+observable+stochastic+domains%2E+Artificial+Intelligence%2C+101%281%29%3A99%E2%80%93134%2C+May+1998%2E+ISSN+0004%2D3702%2E+d)
28. Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinitsky. Gpudrive: Data-driven, multi-agent driving simulation at 1 million fps. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2408.01584. [↗](https://arxiv.org/abs/2408.01584)
29. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. ViZ-Doom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pages 341–348, Santorini, Greece, Sep 2016. IEEE. URL http://arxiv.org/abs/16… — [arXiv:1605.02097](https://arxiv.org/abs/1605.02097) [↗](http://arxiv.org/abs/16…)
30. Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning. In Advances in Neural Information Processing Systems, 2023. [find](https://scholar.google.com/scholar?q=Sotetsu+Koyamada%2C+Shinri+Okano%2C+Soichiro+Nishimori%2C+Yu+Murata%2C+Keigo+Habara%2C+Haruka+Kita%2C+and+Shin+Ishii%2E+Pgx%3A+Hardware%2Daccelerated+parallel+game+simulators+for+reinforcement+learning%2E+In+Advances+in+)
31. Edouard Leurent. An environment for autonomous driving decision-making. [https://github.com/eleurent/highway-env](https://github.com/eleurent/highway-env), 2018. [↗](https://github.com/eleurent/highway-env)
32. Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pages 3053–3062. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Eric+Liang%2C+Richard+Liaw%2C+Robert+Nishihara%2C+Philipp+Moritz%2C+Roy+Fox%2C+Ken+Goldberg%2C+Joseph+Gonzalez%2C+Michael+Jordan%2C+and+Ion+Stoica%2E+Rllib%3A+Abstractions+for+distributed+reinforcement+learning%2E+In+Inter)
33. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. [Reinforcement learning on web interfaces using workflow-guided exploration](https://md.research.elamur.ai/free/en/papers/liu-2018-reinforcement-learning-web-interfaces-using/). In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1802.08802. [↗](https://arxiv.org/abs/1802.08802)
34. Mikel Malagón, Josu Ceberio, and Jose A. Lozano. Craftium: An extensible framework for creating reinforcement learning environments. arXiv preprint arXiv:2407.03969, 2024. — [arXiv:2407.03969](https://arxiv.org/abs/2407.03969) [find](https://scholar.google.com/scholar?q=Mikel+Malag%C3%B3n%2C+Josu+Ceberio%2C+and+Jose+A%2E+Lozano%2E+Craftium%3A+An+extensible+framework+for+creating+reinforcement+learning+environments%2E+arXiv+preprint+arXiv%3A2407%2E03969%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E03969%5D%28https%3A%2F)
35. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Alex+Graves%2C+Ioannis+Antonoglou%2C+Daan+Wierstra%2C+and+Martin+Riedmiller%2E+Playing+atari+with+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1312%2E5602%2C+)
36. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Andrei+A%2E+Rusu%2C+Joel+Veness%2C+Marc+G%2E+Bellemare%2C+Alex+Graves%2C+Martin+Riedmiller%2C+Andreas+K%2E+Fidjeland%2C+Georg+Ostrovski%2C+Stig+Petersen%2C+Charles+Beattie%2C+)
37. Alistair Muldal, Yotam Doron, John Aslanides, Tim Harley, Tom Ward, and Siqi Liu. dm_env: A python interface for reinforcement learning environments, 2019. URL [http://github.com/](http://github.com/) deepmind/dm_env. [↗](http://github.com/)
38. Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado Van Hasselt. Behaviour suite for reinforcement learning, 2020. URL https://arxiv.org/abs/19… — [arXiv:1908.03568](https://arxiv.org/abs/1908.03568) [↗](https://arxiv.org/abs/19…)
39. Mathieu Poliquin. Stable retro, a maintained fork of openai’s gym-retro. [https://github.com/](https://github.com/) Farama-Foundation/stable-retro, 2025. [↗](https://github.com/)
40. Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331–434, 1990. [find](https://scholar.google.com/scholar?q=Martin+L+Puterman%2E+Markov+decision+processes%2E+Handbooks+in+operations+research+and+management+science%2C+2%3A331%E2%80%93434%2C+1990%2E)
41. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL [http://jmlr.org/papers/v22/20-1364.html](http://jmlr.org/papers/v22/20-1364.html). [↗](http://jmlr.org/papers/v22/20-1364.html)
42. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs], August 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+Policy+Optimization+Algorithms%2E+arXiv%3A1707%2E06347+%5Bcs%5D%2C+August+2017%2E)
43. Yang Shao, Quan Kong, Tadayuki Matsumura, Taiki Fuji, Kiyoto Ito, and Hiroyuki Mizuno. Mask atari for deep reinforcement learning as pomdp benchmarks, 2022. URL [https://arxiv.org/](https://arxiv.org/) abs/2203.16777. — [arXiv:2203.16777](https://arxiv.org/abs/2203.16777) [↗](https://arxiv.org/)
44. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Julian+Schrittwieser%2C+Karen+Simonyan%2C+Ioannis+Antonoglou%2C+Aja+Huang%2C+Arthur+Guez%2C+Thomas+Hubert%2C+Lucas+Baker%2C+Matthew+Lai%2C+Adrian+Bolton%2C+et+al%2E+Mastering+the+game+of+go+without+human+kn)
45. Joseph Suarez. Pufferlib: Making reinforcement learning libraries and environments play nice, 2024. URL https://arxiv.org/abs/2406.12905. — [arXiv:2406.12905](https://arxiv.org/abs/2406.12905) [↗](https://arxiv.org/abs/2406.12905)
46. Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, pages 1057–1063, Cambridge, MA, USA, Novembe… [find](https://scholar.google.com/scholar?q=Richard+S%2E+Sutton%2C+David+McAllester%2C+Satinder+Singh%2C+and+Yishay+Mansour%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+In+Proceedings+of+the+12th+International+Confer)
47. Jun Jet Tai, Jim Wong, Mauro Innocente, Nadjim Horri, James Brusey, and Swee King Phang. Pyflyt– uav simulation environments for reinforcement learning research. arXiv preprint arXiv:2304.01305, 2023. — [arXiv:2304.01305](https://arxiv.org/abs/2304.01305) [find](https://scholar.google.com/scholar?q=Jun+Jet+Tai%2C+Jim+Wong%2C+Mauro+Innocente%2C+Nadjim+Horri%2C+James+Brusey%2C+and+Swee+King+Phang%2E+Pyflyt%E2%80%93+uav+simulation+environments+for+reinforcement+learning+research%2E+arXiv+preprint+arXiv%3A2304%2E01305%2C+2023%2E)
48. J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 15032–15043, 2021… [find](https://scholar.google.com/scholar?q=J+Terry%2C+Benjamin+Black%2C+Nathaniel+Grammel%2C+Mario+Jayakumar%2C+Ananth+Hari%2C+Ryan+Sullivan%2C+Luis+S+Santos%2C+Clemens+Dieffendahl%2C+Caroline+Horsch%2C+Rodrigo+Perez%2DVicente%2C+et+al%2E+Pettingzoo%3A+Gym+for+multi%2Dag)
49. J. K. Terry, Benjamin Black, and Luis Santos. Multiplayer support for the arcade learning environment, 2021b. URL https://arxiv.org/abs/2009.09341. — [arXiv:2009.09341](https://arxiv.org/abs/2009.09341) [↗](https://arxiv.org/abs/2009.09341)
50. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. [find](https://scholar.google.com/scholar?q=Emanuel+Todorov%2C+Tom+Erez%2C+and+Yuval+Tassa%2E+Mujoco%3A+A+physics+engine+for+model%2Dbased+control%2E+In+2012+IEEE%2FRSJ+International+Conference+on+Intelligent+Robots+and+Systems%2C+pages+5026%E2%80%935033%2E)
51. Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.org/ 10.1016/j.simpa.2020.10… — [doi:10.1016/j.simpa.2020.100022](https://doi.org/10.1016/j.simpa.2020.100022) [find](https://scholar.google.com/scholar?q=Saran+Tunyasuvunakool%2C+Alistair+Muldal%2C+Yotam+Doron%2C+Siqi+Liu%2C+Steven+Bohez%2C+Josh+Merel%2C+Tom+Erez%2C+Timothy+Lillicrap%2C+Nicolas+Heess%2C+and+Yuval+Tassa%2E+dm%5Fcontrol%3A+Software+and+tasks+for+continuous+cont)
52. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Igor+Babuschkin%2C+Wojciech+M+Czarnecki%2C+Micha%C3%ABl+Mathieu%2C+Andrew+Dudzik%2C+Junyoung+Chung%2C+David+H+Choi%2C+Richard+Powell%2C+Timo+Ewalds%2C+Petko+Georgiev%2C+et+al%2E+Grandmaster+level+in+starcraft+i)
53. Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Hang Su, and Jun Zhu. Tianshou: a Highly Modularized Deep Reinforcement Learning Library. arXiv:2107.14171 [cs], July 2021. URL http://arxiv.org/abs/2107.14171. arXiv: 2107.14171. — [arXiv:2107.14171](https://arxiv.org/abs/2107.14171) [↗](http://arxiv.org/abs/2107.14171)
54. Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. En-vPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho… [find](https://scholar.google.com/scholar?q=Jiayi+Weng%2C+Min+Lin%2C+Shengyi+Huang%2C+Bo+Liu%2C+Denys+Makoviichuk%2C+Viktor+Makoviychuk%2C+Zichen+Liu%2C+Yufan+Song%2C+Ting+Luo%2C+Yukun+Jiang%2C+Zhongwen+Xu%2C+and+Shuicheng+Yan%2E+En%2DvPool%3A+A+highly+parallel+reinforcem)
55. Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments, 2019. URL https://arxiv.org/abs/1903.03176. — [arXiv:1903.03176](https://arxiv.org/abs/1903.03176) [↗](https://arxiv.org/abs/1903.03176)
56. Omar G. Younis, Rodrigo Perez-Vicente, John U. Balis, Will Dudley, Alex Davey, and Jordan K Terry. Minari, September 2024. URL https://doi.org/10.5281/zenodo.13767625. — [doi:10.5281/zenodo.13767625](https://doi.org/10.5281/zenodo.13767625) [↗](https://doi.org/10.5281/zenodo.13767625)
57. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https://arxiv.org/abs/1910.10897. — [arXiv:1910.10897](https://arxiv.org/abs/1910.10897) [↗](https://arxiv.org/abs/1910.10897)

*Updated 2 November 2026*