---
title: "Deep Reinforcement Learning at the Edge of the Statistical"
description: "As an alternative, they propose three tools: interval estimates via stratified bootstrap, performance… 116 references, cited by 49 papers here."
authors: ["Rishabh Agarwal", "Max Schwarzer", "Pablo Samuel Castro", "Aaron Courville", "Marc G. Bellemare"]
year: 2021
venue: "arXiv:2108.13264"
source: "https://arxiv.org/abs/2108.13264"
doi: "10.48550/arxiv.2108.13264"
arxiv: "2108.13264"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agarwal-2021-deep-reinforcement-learning-edge-statistical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agarwal-2021-deep-reinforcement-learning-edge-statistical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / Deep Reinforcement Learning at the Edge of the Statistical Precipice

# Deep Reinforcement Learning at the Edge of the Statistical Precipice

**Authors:** [Rishabh Agarwal](https://md.research.elamur.ai/free/en/authors/rishabh-agarwal/), Max Schwarzer, Pablo Samuel Castro, [Aaron Courville](https://md.research.elamur.ai/free/en/authors/aaron-courville/) and [Marc G. Bellemare](https://md.research.elamur.ai/free/en/authors/marc-g-bellemare/) · **Year:** 2021 · **Venue:** arXiv:2108.13264 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2108.13264) · [PDF](https://arxiv.org/pdf/2108.13264) · [DOI 10.48550/arxiv.2108.13264](https://doi.org/10.48550/arxiv.2108.13264) · [arXiv 2108.13264](https://arxiv.org/abs/2108.13264) · [OpenAlex](https://openalex.org/W3197485521)

**Citations:** 41 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/agarwal-2021-deep-reinforcement-learning-edge-statistical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors examine how reliably algorithms are compared in deep RL when only a few runs per task are available. Because computation is expensive, researchers typically perform 3–10 runs and report only point estimates: the mean or median of the normalized score. The randomness across runs is not taken into account. Using the Atari 100k benchmark as an example, the authors show that such numbers can deviate substantially from the expected values. As a result, conclusions about which method is better are sometimes wrong. As an alternative, they propose three tools: interval estimates via stratified bootstrap, performance profiles (score distributions across all runs), and more robust aggregate metrics, primarily the interquartile mean (IQM). With this toolkit, the authors reanalyze results on ALE, Procgen, and the DeepMind Control Suite. There, too, discrepancies with earlier comparisons emerge. All recommendations are implemented in the open-source library rliable. The work is important because it provides a practical methodology for reliable evaluation without requiring hundreds of runs.

## Key points
- The paper received the Outstanding Paper Award at NeurIPS 2021. The rliable library is available on GitHub together with a Colab notebook, and the raw data of individual runs is published at `gs://rl-benchmark-data`.
- Evaluating 5 runs on more than 50 Atari 2600 games under the standard protocol requires more than 1000 GPU-days. Therefore, 3–10 runs predominate in practice.
- The median remains highly uncertain even with 50 runs and does not change if almost half of the tasks have a score of zero. The mean is easily distorted by outlier tasks. IQM is computed over the middle 50 % of all runs, so it is robust to outliers and yields narrower confidence intervals.
- The authors recommend additionally reporting the average probability of improvement and the optimality gap.
- On Procgen, UCB-DrAC outperforms PLR with a probability of only 40–50 %. Many improvements claimed in the literature hold with a probability of only 50–70 %.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville and Marc G. Bellemare. 2021. Deep Reinforcement Learning at the Edge of the Statistical Precipice. arXiv:2108.13264.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{agarwal2021deep,
  title = {{Deep Reinforcement Learning at the Edge of the Statistical Precipice}},
  author = {Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville and Marc G. Bellemare},
  year = {2021},
  doi = {10.48550/arxiv.2108.13264},
  eprint = {2108.13264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.13264}
}
```

- [← previous BERT Busters: Outlier Dimensions that Disrupt Transformers](https://md.research.elamur.ai/free/en/papers/kovaleva-2021-bert-busters-outlier-dimensions-that/)
- [next → Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://md.research.elamur.ai/free/en/papers/kim-2021-perspective-taking-pragmatics-generating-empathetic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/)

## References

116 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020. [find](https://scholar.google.com/scholar?q=Rishabh+Agarwal%2C+Dale+Schuurmans%2C+and+Mohammad+Norouzi%2E+An+optimistic+perspective+on+offline+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning%2C+2020%2E)
2. Valentin Amrhein, Sander Greenland, and Blake McShane. Scientists rise up against statistical significance. Nature, 2019. [find](https://scholar.google.com/scholar?q=Valentin+Amrhein%2C+Sander+Greenland%2C+and+Blake+McShane%2E+Scientists+rise+up+against+statistical+significance%2E+Nature%2C+2019%2E)
3. Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507–517. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Adri%C3%A0+Puigdom%C3%A8nech+Badia%2C+Bilal+Piot%2C+Steven+Kapturowski%2C+Pablo+Sprechmann%2C+Alex+Vitvitskyi%2C+Zhaohan+Daniel+Guo%2C+and+Charles+Blundell%2E+Agent57%3A+Outperforming+the+atari+human+benchmark%2E+In+Internationa)
4. Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature News, 2016. [find](https://scholar.google.com/scholar?q=Monya+Baker%2E+1%2C500+scientists+lift+the+lid+on+reproducibility%2E+Nature+News%2C+2016%2E)
5. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. [The arcade learning environment: An evaluation platform for general agents](https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/). Journal of Artificial Intelligence Research, 47:253–279, 2013.
6. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Marc+G+Bellemare%2C+Will+Dabney%2C+and+R%C3%A9mi+Munos%2E+A+distributional+perspective+on+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning%2C+pages+449%E2%80%93458%2E+PMLR%2C+2017%2E)
7. Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 2020. [find](https://scholar.google.com/scholar?q=Marc+G+Bellemare%2C+Salvatore+Candido%2C+Pablo+Samuel+Castro%2C+Jun+Gong%2C+Marlos+C+Machado%2C+Subhodeep+Moitra%2C+Sameera+S+Ponda%2C+and+Ziyu+Wang%2E+Autonomous+navigation+of+stratospheric+balloons+using+reinforcem)
8. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. [find](https://scholar.google.com/scholar?q=Christopher+Berner%2C+Greg+Brockman%2C+Brooke+Chan%2C+Vicki+Cheung%2C+Przemys%C5%82aw+D%CB%9Bebiak%2C+Christy+Dennison%2C+David+Farhi%2C+Quirin+Fischer%2C+Shariq+Hashme%2C+Chris+Hesse%2C+et+al%2E+Dota+2+with+large+scale+deep+reinfor)
9. Peter J Bickel, Friedrich Götze, and Willem R van Zwet. Resampling fewer than n observations: gains, losses, and remedies for losses. In Selected works of Willem van Zwet, pages 267–297. Springer, 2012. [find](https://scholar.google.com/scholar?q=Peter+J+Bickel%2C+Friedrich+G%C3%B6tze%2C+and+Willem+R+van+Zwet%2E+Resampling+fewer+than+n+observations%3A+gains%2C+losses%2C+and+remedies+for+losses%2E+In+Selected+works+of+Willem+van+Zwet%2C+pages+267%E2%80%93297%2E+Springer%2C+201)
10. Mauro Birattari and Marco Dorigo. How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs? Optimization letters, 2007. [find](https://scholar.google.com/scholar?q=Mauro+Birattari+and+Marco+Dorigo%2E+How+to+assess+and+report+the+performance+of+a+stochastic+algorithm+on+a+benchmark+problem%3A+mean+or+best+result+on+a+number+of+runs%3F+Optimization+letters%2C+2007%2E)
11. Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research is reproducible. In International Conference on Machine Learning, pages 725–734, 2019. [find](https://scholar.google.com/scholar?q=Xavier+Bouthillier%2C+C%C3%A9sar+Laurent%2C+and+Pascal+Vincent%2E+Unreproducible+research+is+reproducible%2E+In+International+Conference+on+Machine+Learning%2C+pages+725%E2%80%93734%2C+2019%2E)
12. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3, 2021. [find](https://scholar.google.com/scholar?q=Xavier+Bouthillier%2C+Pierre+Delaunay%2C+Mirko+Bronzi%2C+Assya+Trofimov%2C+Brennan+Nichyporuk%2C+Justin+Szeto%2C+Nazanin+Mohammadi+Sepahvand%2C+Edward+Raff%2C+Kanika+Madan%2C+Vikram+Voleti%2C+et+al%2E+Accounting+for+varian)
13. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/google/](http://github.com/google/) jax. [↗](http://github.com/google/)
14. Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018. — [arXiv:1812.06110](https://arxiv.org/abs/1812.06110) [find](https://scholar.google.com/scholar?q=Pablo+Samuel+Castro%2C+Subhodeep+Moitra%2C+Carles+Gelada%2C+Saurabh+Kumar%2C+and+Marc+G+Bellemare%2E+Dopamine%3A+A+research+framework+for+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1812%2E06110%2C+2018%2E+%E2%80%94+%5BarX)
15. Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Johan+Samir+Obando+Ceron+and+Pablo+Samuel+Castro%2E+Revisiting+rainbow%3A+Promoting+more+insightful+and+inclusive+deep+reinforcement+learning+research%2E+In+International+Conference+on+Machine+Learning%2C+202)
16. Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Stephanie+CY+Chan%2C+Samuel+Fishman%2C+Anoop+Korattikara%2C+John+Canny%2C+and+Sergio+Guadarrama%2E+Measuring+the+reliability+of+reinforcement+learning+algorithms%2E+In+International+Conference+on+Learning+Represe)
17. Kaleigh Clary, Emma Tosch, John Foley, and David Jensen. Let’s play again: Variability of deep reinforcement learning agents in atari environments. arXiv preprint arXiv:1904.06312, 2019. — [arXiv:1904.06312](https://arxiv.org/abs/1904.06312) [find](https://scholar.google.com/scholar?q=Kaleigh+Clary%2C+Emma+Tosch%2C+John+Foley%2C+and+David+Jensen%2E+Let%E2%80%99s+play+again%3A+Variability+of+deep+reinforcement+learning+agents+in+atari+environments%2E+arXiv+preprint+arXiv%3A1904%2E06312%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904)
18. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Chris+Hesse%2C+Jacob+Hilton%2C+and+John+Schulman%2E+Leveraging+procedural+generation+to+benchmark+reinforcement+learning%2E+In+International+conference+on+machine+learning%2C+pages+2048%E2%80%932056%2E+PMLR%2C+)
19. Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. arXiv preprint arXiv:2009.04416, 2020. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Jacob+Hilton%2C+Oleg+Klimov%2C+and+John+Schulman%2E+Phasic+policy+gradient%2E+arXiv+preprint+arXiv%3A2009%2E04416%2C+2020%2E)
20. Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv:1806.08295, 2018. [find](https://scholar.google.com/scholar?q=C%C3%A9dric+Colas%2C+Olivier+Sigaud%2C+and+Pierre%2DYves+Oudeyer%2E+How+many+random+seeds%3F+statistical+power+analysis+in+deep+reinforcement+learning+experiments%2E+arXiv+preprint+arXiv%3A1806%2E08295%2C+2018%2E)
21. Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. A hitchhiker’s guide to statistical comparisons of reinforcement learning algorithms. arXiv preprint arXiv:1904.06979, 2019. — [arXiv:1904.06979](https://arxiv.org/abs/1904.06979) [find](https://scholar.google.com/scholar?q=C%C3%A9dric+Colas%2C+Olivier+Sigaud%2C+and+Pierre%2DYves+Oudeyer%2E+A+hitchhiker%E2%80%99s+guide+to+statistical+comparisons+of+reinforcement+learning+algorithms%2E+arXiv+preprint+arXiv%3A1904%2E06979%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E06979%5D)
22. Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096–1105. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Will+Dabney%2C+Georg+Ostrovski%2C+David+Silver%2C+and+R%C3%A9mi+Munos%2E+Implicit+quantile+networks+for+distributional+reinforcement+learning%2E+In+International+conference+on+machine+learning%2C+pages+1096%E2%80%931105%2E+PMLR)
23. Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. [find](https://scholar.google.com/scholar?q=Will+Dabney%2C+Mark+Rowland%2C+Marc+G+Bellemare%2C+and+R%C3%A9mi+Munos%2E+Distributional+reinforcement+learning+with+quantile+regression%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2C+2018%2E)
24. Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07002, 2021. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Yi+Tay%2C+Alexey+A+Gritsenko%2C+Zhe+Zhao%2C+Neil+Houlsby%2C+Fernando+Diaz%2C+Donald+Metzler%2C+and+Oriol+Vinyals%2E+The+benchmark+lottery%2E+arXiv+preprint+arXiv%3A2107%2E07002%2C+2021%2E)
25. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+arXiv+pre)
26. Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical programming, 91(2):201–213, 2002. [find](https://scholar.google.com/scholar?q=Elizabeth+D+Dolan+and+Jorge+J+Mor%C3%A9%2E+Benchmarking+optimization+software+with+performance+profiles%2E+Mathematical+programming%2C+91%282%29%3A201%E2%80%93213%2C+2002%2E)
27. Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance-how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+Deep+dominance%2Dhow+to+properly+compare+deep+neural+models%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2019)
28. Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019. [find](https://scholar.google.com/scholar?q=Adrien+Ecoffet%2C+Joost+Huizinga%2C+Joel+Lehman%2C+Kenneth+O+Stanley%2C+and+Jeff+Clune%2E+Go%2Dexplore%3A+a+new+approach+for+hard%2Dexploration+problems%2E+arXiv+preprint+arXiv%3A1901%2E10995%2C+2019%2E)
29. Bradley Efron. Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7:1–26, 1979. [find](https://scholar.google.com/scholar?q=Bradley+Efron%2E+Bootstrap+methods%3A+another+look+at+the+jackknife%2E+The+Annals+of+Statistics%2C+7%3A1%E2%80%9326%2C+1979%2E)
30. Bradley Efron. Better bootstrap confidence intervals. Journal of the American statistical Association, 1987. [find](https://scholar.google.com/scholar?q=Bradley+Efron%2E+Better+bootstrap+confidence+intervals%2E+Journal+of+the+American+statistical+Association%2C+1987%2E)
31. Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503–556, 2005. [find](https://scholar.google.com/scholar?q=Damien+Ernst%2C+Pierre+Geurts%2C+and+Louis+Wehenkel%2E+Tree%2Dbased+batch+mode+reinforcement+learning%2E+Journal+of+Machine+Learning+Research%2C+6%3A503%E2%80%93556%2C+2005%2E)
32. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Lasse+Espeholt%2C+Hubert+Soyer%2C+Remi+Munos%2C+Karen+Simonyan%2C+Vlad+Mnih%2C+Tom+Ward%2C+Yotam+Doron%2C+Vlad+Firoiu%2C+Tim+Harley%2C+Iain+Dunning%2C+et+al%2E+Impala%3A+Scalable+distributed+deep%2Drl+with+importance+weighted+)
33. Gerd Gigerenzer. Statistical rituals: The replication delusion and how we got there. Advances in Methods and Practices in Psychological Science, 1(2):198–218, 2018. [find](https://scholar.google.com/scholar?q=Gerd+Gigerenzer%2E+Statistical+rituals%3A+The+replication+delusion+and+how+we+got+there%2E+Advances+in+Methods+and+Practices+in+Psychological+Science%2C+1%282%29%3A198%E2%80%93218%2C+2018%2E)
34. Steven N Goodman, Daniele Fanelli, and John PA Ioannidis. What does research reproducibility mean? Science translational medicine, 8(341):341ps12–341ps12, 2016. [find](https://scholar.google.com/scholar?q=Steven+N+Goodman%2C+Daniele+Fanelli%2C+and+John+PA+Ioannidis%2E+What+does+research+reproducibility+mean%3F+Science+translational+medicine%2C+8%28341%29%3A341ps12%E2%80%93341ps12%2C+2016%2E)
35. Sander Greenland, Stephen J Senn, Kenneth J Rothman, John B Carlin, Charles Poole, Steven N Goodman, and Douglas G Altman. Statistical tests, p values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology, 2016. [find](https://scholar.google.com/scholar?q=Sander+Greenland%2C+Stephen+J+Senn%2C+Kenneth+J+Rothman%2C+John+B+Carlin%2C+Charles+Poole%2C+Steven+N+Goodman%2C+and+Douglas+G+Altman%2E+Statistical+tests%2C+p+values%2C+confidence+intervals%2C+and+power%3A+a+guide+to+misi)
36. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Tuomas+Haarnoja%2C+Aurick+Zhou%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Soft+actor%2Dcritic%3A+Off%2Dpolicy+maximum+entropy+deep+reinforcement+learning+with+a+stochastic+actor%2E+In+International+Conference+on+Machin)
37. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Timothy+Lillicrap%2C+Jimmy+Ba%2C+and+Mohammad+Norouzi%2E+Dream+to+control%3A+Learning+behaviors+by+latent+imagination%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
38. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. [find](https://scholar.google.com/scholar?q=Danijar+Hafner%2C+Timothy+Lillicrap%2C+Mohammad+Norouzi%2C+and+Jimmy+Ba%2E+Mastering+atari+with+discrete+world+models%2E+arXiv+preprint+arXiv%3A2010%2E02193%2C+2020%2E)
39. Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Steven+Hansen%2C+Will+Dabney%2C+Andre+Barreto%2C+David+Warde%2DFarley%2C+Tom+Van+de+Wiele%2C+and+Volodymyr+Mnih%2E+Fast+task+inference+with+variational+intrinsic+successor+features%2E+In+International+Conference+on+L)
40. Nathaniel E Helwig. Bootstrap Confidence Intervals, 01 2021. URL [http://users.stat.umn.edu/](http://users.stat.umn.edu/) ~helwig/notes/npboot-notes.html\#bootstrap-confidence-intervals. [↗](http://users.stat.umn.edu/)
41. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Riashat+Islam%2C+Philip+Bachman%2C+Joelle+Pineau%2C+Doina+Precup%2C+and+David+Meger%2E+Deep+reinforcement+learning+that+matters%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence)
42. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. [find](https://scholar.google.com/scholar?q=Matteo+Hessel%2C+Joseph+Modayil%2C+Hado+Van+Hasselt%2C+Tom+Schaul%2C+Georg+Ostrovski%2C+Will+Dabney%2C+Dan+Horgan%2C+Bilal+Piot%2C+Mohammad+Azar%2C+and+David+Silver%2E+Rainbow%3A+Combining+improvements+in+deep+reinforcemen)
43. Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado van Hasselt. Muesli: Combining improvements in policy optimization. Proceedings of the 38th International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Matteo+Hessel%2C+Ivo+Danihelka%2C+Fabio+Viola%2C+Arthur+Guez%2C+Simon+Schmitt%2C+Laurent+Sifre%2C+Theophane+Weber%2C+David+Silver%2C+and+Hado+van+Hasselt%2E+Muesli%3A+Combining+improvements+in+policy+optimization%2E+Procee)
44. John PA Ioannidis. Why most published research findings are false. PLoS medicine, 2(8):e124, 2005. [find](https://scholar.google.com/scholar?q=John+PA+Ioannidis%2E+Why+most+published+research+findings+are+false%2E+PLoS+medicine%2C+2%288%29%3Ae124%2C+2005%2E)
45. Alex Irpan. Deep reinforcement learning doesn’t work yet. [https://www.alexirpan.com/2018/02/](https://www.alexirpan.com/2018/02/) 14/rl-hard.html, 2018. [↗](https://www.alexirpan.com/2018/02/)
46. Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133, 2017. — [arXiv:1708.04133](https://arxiv.org/abs/1708.04133) [find](https://scholar.google.com/scholar?q=Riashat+Islam%2C+Peter+Henderson%2C+Maziar+Gomrokchi%2C+and+Doina+Precup%2E+Reproducibility+of+benchmarked+deep+reinforcement+learning+tasks+for+continuous+control%2E+arXiv+preprint+arXiv%3A1708%2E04133%2C+2017%2E+%E2%80%94+%5Ba)
47. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. [Reinforcement learning with unsupervised auxiliary tasks](https://md.research.elamur.ai/free/en/papers/jaderberg-2016-reinforcement-learning-unsupervised-auxiliary-tasks/). arXiv preprint arXiv:1611.05397, 2016.
48. Minqi Jiang, Ed Grefenstette, and Tim Rocktäschel. Prioritized level replay. International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Minqi+Jiang%2C+Ed+Grefenstette%2C+and+Tim+Rockt%C3%A4schel%2E+Prioritized+level+replay%2E+International+Conference+on+Machine+Learning%2C+2021%2E)
49. Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning, pages 4962–4973. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Scott+Jordan%2C+Yash+Chandak%2C+Daniel+Cohen%2C+Mengxue+Zhang%2C+and+Philip+Thomas%2E+Evaluating+the+performance+of+reinforcement+learning+algorithms%2E+In+International+Conference+on+Machine+Learning%2C+pages+4962)
50. Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019. [find](https://scholar.google.com/scholar?q=Lukasz+Kaiser%2C+Mohammad+Babaeizadeh%2C+Piotr+Milos%2C+Blazej+Osinski%2C+Roy+H+Campbell%2C+Konrad+Czechowski%2C+Dumitru+Erhan%2C+Chelsea+Finn%2C+Piotr+Kozakowski%2C+Sergey+Levine%2C+et+al%2E+Model%2Dbased+reinforcement+lear)
51. Kacper Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003.10181, 2020. — [arXiv:2003.10181](https://arxiv.org/abs/2003.10181) [find](https://scholar.google.com/scholar?q=Kacper+Kielak%2E+Do+recent+advancements+in+model%2Dbased+deep+reinforcement+learning+really+improve+data+efficiency%3F+arXiv+preprint+arXiv%3A2003%2E10181%2C+2020%2E+%E2%80%94+%5BarXiv%3A2003%2E10181%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E)
52. Sergey Kolesnikov and Oleksii Hrinchuk. Catalyst. rl: a distributed framework for reproducible rl research. arXiv preprint arXiv:1903.00027, 2019. — [arXiv:1903.00027](https://arxiv.org/abs/1903.00027) [find](https://scholar.google.com/scholar?q=Sergey+Kolesnikov+and+Oleksii+Hrinchuk%2E+Catalyst%2E+rl%3A+a+distributed+framework+for+reproducible+rl+research%2E+arXiv+preprint+arXiv%3A1903%2E00027%2C+2019%2E+%E2%80%94+%5BarXiv%3A1903%2E00027%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1903%2E00027)
53. Ilya Kostrikov*, Denis Yarats*, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Ilya+Kostrikov%2A%2C+Denis+Yarats%2A%2C+and+Rob+Fergus%2E+Image+augmentation+is+all+you+need%3A+Regularizing+deep+reinforcement+learning+from+pixels%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
54. Piotr Kozakowski, Lukasz Kaiser, Henryk Michalewski, Afroz Mohiuddin, and Katarzyna Kańska. Qvalue weighted regression: Reinforcement learning with limited data. arXiv preprint arXiv:2102.06782, 2021. [find](https://scholar.google.com/scholar?q=Piotr+Kozakowski%2C+Lukasz+Kaiser%2C+Henryk+Michalewski%2C+Afroz+Mohiuddin%2C+and+Katarzyna+Kan%CC%81ska%2E+Qvalue+weighted+regression%3A+Reinforcement+learning+with+limited+data%2E+arXiv+preprint+arXiv%3A2102%2E06782%2C+2021)
55. Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. NeurIPS, 32:10724–10734, 2019. [find](https://scholar.google.com/scholar?q=Tejas+D+Kulkarni%2C+Ankush+Gupta%2C+Catalin+Ionescu%2C+Sebastian+Borgeaud%2C+Malcolm+Reynolds%2C+Andrew+Zisserman%2C+and+Volodymyr+Mnih%2E+Unsupervised+learning+of+object+keypoints+for+perception+and+control%2E+NeurI)
56. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020. [find](https://scholar.google.com/scholar?q=Michael+Laskin%2C+Aravind+Srinivas%2C+and+Pieter+Abbeel%2E+Curl%3A+Contrastive+unsupervised+representations+for+reinforcement+learning%2E+In+International+Conference+on+Machine+Learning%2C+2020%2E)
57. Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Misha+Laskin%2C+Kimin+Lee%2C+Adam+Stooke%2C+Lerrel+Pinto%2C+Pieter+Abbeel%2C+and+Aravind+Srinivas%2E+Reinforcement+learning+with+augmented+data%2E+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E)
58. Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33, 2020. [find](https://scholar.google.com/scholar?q=Alex+Lee%2C+Anusha+Nagabandi%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Stochastic+latent+actor%2Dcritic%3A+Deep+reinforcement+learning+with+a+latent+variable+model%2E+Advances+in+Neural+Information+Processing+System)
59. Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Kimin+Lee%2C+Michael+Laskin%2C+Aravind+Srinivas%2C+and+Pieter+Abbeel%2E+Sunrise%3A+A+simple+unified+framework+for+ensemble+learning+in+deep+reinforcement+learning%2E+International+Conference+on+Machine+Learning%2C+)
60. Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=Kuang%2DHuei+Lee%2C+Ian+Fischer%2C+Anthony+Liu%2C+Yijie+Guo%2C+Honglak+Lee%2C+John+Canny%2C+and+Sergio+Guadarrama%2E+Predictive+information+accelerates+learning+in+rl%2E+Advances+in+Neural+Information+Processing+System)
61. Haim Levy. Stochastic dominance and expected utility: Survey and analysis. Management science, 38(4): 555–593, 1992. [find](https://scholar.google.com/scholar?q=Haim+Levy%2E+Stochastic+dominance+and+expected+utility%3A+Survey+and+analysis%2E+Management+science%2C+38%284%29%3A+555%E2%80%93593%2C+1992%2E)
62. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. [find](https://scholar.google.com/scholar?q=Timothy+P+Lillicrap%2C+Jonathan+J+Hunt%2C+Alexander+Pritzel%2C+Nicolas+Heess%2C+Tom+Erez%2C+Yuval+Tassa%2C+David+Silver%2C+and+Daan+Wierstra%2E+Continuous+control+with+deep+reinforcement+learning%2E+arXiv+preprint+arXi)
63. Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Significant improvements over the state of the art? a case study of the ms marco document ranking leaderboard. arXiv preprint arXiv:2102.12887, 2021. — [arXiv:2102.12887](https://arxiv.org/abs/2102.12887) [find](https://scholar.google.com/scholar?q=Jimmy+Lin%2C+Daniel+Campos%2C+Nick+Craswell%2C+Bhaskar+Mitra%2C+and+Emine+Yilmaz%2E+Significant+improvements+over+the+state+of+the+art%3F+a+case+study+of+the+ms+marco+document+ranking+leaderboard%2E+arXiv+preprint+)
64. Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Li Jian, Nenghai Yu, and Tie-Yan Liu. Return-based contrastive representation learning for reinforcement learning. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Guoqing+Liu%2C+Chuheng+Zhang%2C+Li+Zhao%2C+Tao+Qin%2C+Jinhua+Zhu%2C+Li+Jian%2C+Nenghai+Yu%2C+and+Tie%2DYan+Liu%2E+Return%2Dbased+contrastive+representation+learning+for+reinforcement+learning%2E+In+International+Conference)
65. Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04551, 2021. — [arXiv:2103.04551](https://arxiv.org/abs/2103.04551) [find](https://scholar.google.com/scholar?q=Hao+Liu+and+Pieter+Abbeel%2E+Behavior+from+the+void%3A+Unsupervised+active+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2103%2E04551%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E04551%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2103%2E04551%29)
66. Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Hao+Liu+and+Pieter+Abbeel%2E+Aps%3A+Active+pretraining+with+successor+features%2E+In+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+2021%2E)
67. Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. arXiv preprint arXiv:1711.10337, 2017. [find](https://scholar.google.com/scholar?q=Mario+Lucic%2C+Karol+Kurach%2C+Marcin+Michalski%2C+Sylvain+Gelly%2C+and+Olivier+Bousquet%2E+Are+gans+created+equal%3F+a+large%2Dscale+study%2E+arXiv+preprint+arXiv%3A1711%2E10337%2C+2017%2E)
68. Nicolai A Lynnerup, Laura Nolling, Rasmus Hasle, and John Hallam. A survey on reproducibility by evaluating deep reinforcement learning algorithms on real-world robots. In Conference on Robot Learning, 2020. [find](https://scholar.google.com/scholar?q=Nicolai+A+Lynnerup%2C+Laura+Nolling%2C+Rasmus+Hasle%2C+and+John+Hallam%2E+A+survey+on+reproducibility+by+evaluating+deep+reinforcement+learning+algorithms+on+real%2Dworld+robots%2E+In+Conference+on+Robot+Learning)
69. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. [Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents](https://md.research.elamur.ai/free/en/papers/machado-2017-revisiting-arcade-learning-environment-evaluation/). Journal of Artificial Intelligence Research, 2018.
70. Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018. — [arXiv:1803.07055](https://arxiv.org/abs/1803.07055) [find](https://scholar.google.com/scholar?q=Horia+Mania%2C+Aurelia+Guy%2C+and+Benjamin+Recht%2E+Simple+random+search+provides+a+competitive+approach+to+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1803%2E07055%2C+2018%2E+%E2%80%94+%5BarXiv%3A1803%2E07055%5D%28https%3A%2F%2Farxiv%2E)
71. Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60, 1947. [find](https://scholar.google.com/scholar?q=Henry+B+Mann+and+Donald+R+Whitney%2E+On+a+test+of+whether+one+of+two+random+variables+is+stochastically+larger+than+the+other%2E+The+annals+of+mathematical+statistics%2C+pages+50%E2%80%9360%2C+1947%2E)
72. Muhammad Rizki Maulana and Wee Sun Lee. Ensemble and auxiliary tasks for data-efficient deep reinforcement learning. arXiv preprint arXiv:2107.01904, 2021. — [arXiv:2107.01904](https://arxiv.org/abs/2107.01904) [find](https://scholar.google.com/scholar?q=Muhammad+Rizki+Maulana+and+Wee+Sun+Lee%2E+Ensemble+and+auxiliary+tasks+for+data%2Defficient+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2107%2E01904%2C+2021%2E+%E2%80%94+%5BarXiv%3A2107%2E01904%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
73. Blakeley B McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L Tackett. Abandon statistical significance. The American Statistician, 2019. [find](https://scholar.google.com/scholar?q=Blakeley+B+McShane%2C+David+Gal%2C+Andrew+Gelman%2C+Christian+Robert%2C+and+Jennifer+L+Tackett%2E+Abandon+statistical+significance%2E+The+American+Statistician%2C+2019%2E)
74. Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=G%C3%A1bor+Melis%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+On+the+state+of+the+art+of+evaluation+in+neural+language+models%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E)
75. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Andrei+A+Rusu%2C+Joel+Veness%2C+Marc+G+Bellemare%2C+Alex+Graves%2C+Martin+Riedmiller%2C+Andreas+K+Fidjeland%2C+Georg+Ostrovski%2C+et+al%2E+Human%2Dlevel+control+through+)
76. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Adria+Puigdomenech+Badia%2C+Mehdi+Mirza%2C+Alex+Graves%2C+Timothy+Lillicrap%2C+Tim+Harley%2C+David+Silver%2C+and+Koray+Kavukcuoglu%2E+Asynchronous+methods+for+deep+reinforcement+learning%2E+In+Interna)
77. Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic implementations for reproducibility in deep reinforcement learning. arXiv preprint arXiv:1809.05676, 2018. — [arXiv:1809.05676](https://arxiv.org/abs/1809.05676) [find](https://scholar.google.com/scholar?q=Prabhat+Nagarajan%2C+Garrett+Warnell%2C+and+Peter+Stone%2E+Deterministic+implementations+for+reproducibility+in+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1809%2E05676%2C+2018%2E+%E2%80%94+%5BarXiv%3A1809%2E05676%5D%28https)
78. Harold Pashler and Eric-Jan Wagenmakers. Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on psychological science, 7(6):528–530, 2012. [find](https://scholar.google.com/scholar?q=Harold+Pashler+and+Eric%2DJan+Wagenmakers%2E+Editors%E2%80%99+introduction+to+the+special+section+on+replicability+in+psychological+science%3A+A+crisis+of+confidence%3F+Perspectives+on+psychological+science%2C+7%286%29%3A528)
79. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). arXiv preprint arXiv:2003.12206, 2020. [find](https://scholar.google.com/scholar?q=Joelle+Pineau%2C+Philippe+Vincent%2DLamarre%2C+Koustuv+Sinha%2C+Vincent+Larivi%C3%A8re%2C+Alina+Beygelzimer%2C+Florence+d%E2%80%99Alch%C3%A9+Buc%2C+Emily+Fox%2C+and+Hugo+Larochelle%2E+Improving+reproducibility+in+machine+learning+resear)
80. Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Roberta+Raileanu+and+Rob+Fergus%2E+Decoupling+value+and+policy+for+generalization+in+reinforcement+learning%2E+International+Conference+on+Machine+Learning%2C+2021%2E)
81. Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862, 2020. — [arXiv:2006.12862](https://arxiv.org/abs/2006.12862) [find](https://scholar.google.com/scholar?q=Roberta+Raileanu%2C+Max+Goldstein%2C+Denis+Yarats%2C+Ilya+Kostrikov%2C+and+Rob+Fergus%2E+Automatic+data+augmentation+for+generalization+in+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2006%2E12862%2C+2020%2E+%E2%80%94+%5B)
82. David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021. — [arXiv:2102.12425](https://arxiv.org/abs/2102.12425) [find](https://scholar.google.com/scholar?q=David+Raposo%2C+Sam+Ritter%2C+Adam+Santoro%2C+Greg+Wayne%2C+Theophane+Weber%2C+Matt+Botvinick%2C+Hado+van+Hasselt%2C+and+Francis+Song%2E+Synthetic+returns+for+long%2Dterm+credit+assignment%2E+arXiv+preprint+arXiv%3A2102%2E12)
83. Ben Recht. Benchmarking Machine Learning with Performance Profiles, 03 2018. URL [http://www.argmin.net/2018/03/26/performance-profiles/](http://www.argmin.net/2018/03/26/performance-profiles/). [↗](http://www.argmin.net/2018/03/26/performance-profiles/)
84. Nils Reimers and Iryna Gurevych. [Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging](https://md.research.elamur.ai/free/en/papers/reimers-2017-reporting-score-distributions-makes-difference/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, 2017.
85. Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning, 2017. [find](https://scholar.google.com/scholar?q=Samuel+Ritter%2C+David+GT+Barrett%2C+Adam+Santoro%2C+and+Matt+M+Botvinick%2E+Cognitive+psychology+for+deep+neural+networks%3A+A+shape+bias+case+study%2E+In+International+conference+on+machine+learning%2C+2017%2E)
86. Jan Robine, Tobias Uelwer, and Stefan Harmeling. Smaller world models for reinforcement learning. arXiv preprint arXiv:2010.05767, 2020. — [arXiv:2010.05767](https://arxiv.org/abs/2010.05767) [find](https://scholar.google.com/scholar?q=Jan+Robine%2C+Tobias+Uelwer%2C+and+Stefan+Harmeling%2E+Smaller+world+models+for+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2010%2E05767%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E05767%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E05767%29)
87. David Romer. In praise of confidence intervals. In AEA Papers and Proceedings, volume 110, pages 55–60, 2020. [find](https://scholar.google.com/scholar?q=David+Romer%2E+In+praise+of+confidence+intervals%2E+In+AEA+Papers+and+Proceedings%2C+volume+110%2C+pages+55%E2%80%9360%2C+2020%2E)
88. Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017. [find](https://scholar.google.com/scholar?q=Tim+Salimans%2C+Jonathan+Ho%2C+Xi+Chen%2C+Szymon+Sidor%2C+and+Ilya+Sutskever%2E+Evolution+strategies+as+a+scalable+alternative+to+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1703%2E03864%2C+2017%2E)
89. Rohan Saphal, Balaraman Ravindran, Dheevatsa Mudigere, Sasikant Avancha, and Bharat Kaul. Seerl: Sample efficient ensemble reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1100–1108, 2021. [find](https://scholar.google.com/scholar?q=Rohan+Saphal%2C+Balaraman+Ravindran%2C+Dheevatsa+Mudigere%2C+Sasikant+Avancha%2C+and+Bharat+Kaul%2E+Seerl%3A+Sample+efficient+ensemble+reinforcement+learning%2E+In+Proceedings+of+the+20th+International+Conference+o)
90. Tom Schaul, Georg Ostrovski, Iurii Kemaev, and Diana Borsa. Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.05347, 2021. — [arXiv:2105.05347](https://arxiv.org/abs/2105.05347) [find](https://scholar.google.com/scholar?q=Tom+Schaul%2C+Georg+Ostrovski%2C+Iurii+Kemaev%2C+and+Diana+Borsa%2E+Return%2Dbased+scaling%3A+Yet+another+normalisation+trick+for+deep+rl%2E+arXiv+preprint+arXiv%3A2105%2E05347%2C+2021%2E+%E2%80%94+%5BarXiv%3A2105%2E05347%5D%28https%3A%2F%2Farxiv)
91. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2020. [find](https://scholar.google.com/scholar?q=Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Thomas+Hubert%2C+Karen+Simonyan%2C+Laurent+Sifre%2C+Simon+Schmitt%2C+Arthur+Guez%2C+Edward+Lockhart%2C+Demis+Hassabis%2C+Thore+Graepel%2C+et+al%2E+Mastering+atari%2C+go%2C+chess+an)
92. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
93. Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Max+Schwarzer%2C+Ankesh+Anand%2C+Rishab+Goel%2C+R+Devon+Hjelm%2C+Aaron+Courville%2C+and+Philip+Bachman%2E+Data%2Defficient+reinforcement+learning+with+self%2Dpredictive+representations%2E+In+International+Conference+on)
94. Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, et al. The multiberts: Bert reproductions for robustness analysis. arXiv preprint arXiv:2106.16163, 2021. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Steve+Yadlowsky%2C+Jason+Wei%2C+Naomi+Saphra%2C+Alexander+D%E2%80%99Amour%2C+Tal+Linzen%2C+Jasmijn+Bastings%2C+Iulia+Turc%2C+Jacob+Eisenstein%2C+Dipanjan+Das%2C+et+al%2E+The+multiberts%3A+Bert+reproductions+for+ro)
95. Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In Proceedings of the 38th International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Younggyo+Seo%2C+Lili+Chen%2C+Jinwoo+Shin%2C+Honglak+Lee%2C+Pieter+Abbeel%2C+and+Kimin+Lee%2E+State+entropy+maximization+with+random+encoders+for+efficient+exploration%2E+In+Proceedings+of+the+38th+International+Con)
96. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+Van+Den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+et+al%2E+Mastering+the+game+)
97. Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163, 2020. — [arXiv:2010.09163](https://arxiv.org/abs/2010.09163) [find](https://scholar.google.com/scholar?q=Samarth+Sinha%2C+Homanga+Bharadhwaj%2C+Aravind+Srinivas%2C+and+Animesh+Garg%2E+D2rl%3A+Deep+dense+architectures+in+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2010%2E09163%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E09163%5D%28https%3A%2F%2Farx)
98. Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136v2, 2020. — [arXiv:2004.04136](https://arxiv.org/abs/2004.04136) [find](https://scholar.google.com/scholar?q=Aravind+Srinivas%2C+Michael+Laskin%2C+and+Pieter+Abbeel%2E+Curl%3A+Contrastive+unsupervised+representations+for+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2004%2E04136v2%2C+2020%2E+%E2%80%94+%5BarXiv%3A2004%2E04136%5D%28https%3A%2F%2Far)
99. Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1): 9–44, 1988. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2E+Learning+to+predict+by+the+methods+of+temporal+differences%2E+Machine+learning%2C+3%281%29%3A+9%E2%80%9344%2C+1988%2E)
100. Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in neural information processing systems, 1996. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2E+Generalization+in+reinforcement+learning%3A+Successful+examples+using+sparse+coarse+coding%2E+Advances+in+neural+information+processing+systems%2C+1996%2E)
101. Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 2nd edition, 2018. [find](https://scholar.google.com/scholar?q=Richard+S%2E+Sutton+and+Andrew+G%2E+Barto%2E+Reinforcement+learning%3A+An+introduction%2E+MIT+Press%2C+2nd+edition%2C+2018%2E)
102. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+Doina+Precup%2C+and+Satinder+Singh%2E+Between+mdps+and+semi%2Dmdps%3A+A+framework+for+temporal+abstraction+in+reinforcement+learning%2E+Artificial+intelligence%2C+1999%2E)
103. István Szita and András Lörincz. Learning tetris using the noisy cross-entropy method. Neural computation, 2006. [find](https://scholar.google.com/scholar?q=Istv%C3%A1n+Szita+and+Andr%C3%A1s+L%C3%B6rincz%2E+Learning+tetris+using+the+noisy+cross%2Dentropy+method%2E+Neural+computation%2C+2006%2E)
104. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. [find](https://scholar.google.com/scholar?q=Yuval+Tassa%2C+Yotam+Doron%2C+Alistair+Muldal%2C+Tom+Erez%2C+Yazhe+Li%2C+Diego+de+Las+Casas%2C+David+Budden%2C+Abbas+Abdolmaleki%2C+Josh+Merel%2C+Andrew+Lefrancq%2C+et+al%2E+Deepmind+control+suite%2E+arXiv+preprint+arXiv%3A180)
105. Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. Is deep reinforcement learning really superhuman on atari? leveling the playing field. arXiv preprint arXiv:1908.04683, 2019. — [arXiv:1908.04683](https://arxiv.org/abs/1908.04683) [find](https://scholar.google.com/scholar?q=Marin+Toromanoff%2C+Emilie+Wirbel%2C+and+Fabien+Moutarde%2E+Is+deep+reinforcement+learning+really+superhuman+on+atari%3F+leveling+the+playing+field%2E+arXiv+preprint+arXiv%3A1908%2E04683%2C+2019%2E+%E2%80%94+%5BarXiv%3A1908%2E04683%5D)
106. John W Tukey. A survey of sampling from contaminated distributions. Contributions to probability and statistics, pages 448–485, 1960. [find](https://scholar.google.com/scholar?q=John+W+Tukey%2E+A+survey+of+sampling+from+contaminated+distributions%2E+Contributions+to+probability+and+statistics%2C+pages+448%E2%80%93485%2C+1960%2E)
107. Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Hado+van+Hasselt%2C+Matteo+Hessel%2C+and+John+Aslanides%2E+When+to+use+parametric+models+in+reinforcement+learning%3F+NeurIPS%2C+2019%2E)
108. Gaël Varoquaux and Veronika Cheplygina. How i failed machine learning in medical imaging– shortcomings and recommendations. arXiv preprint arXiv:2103.10292, 2021. — [arXiv:2103.10292](https://arxiv.org/abs/2103.10292) [find](https://scholar.google.com/scholar?q=Ga%C3%ABl+Varoquaux+and+Veronika+Cheplygina%2E+How+i+failed+machine+learning+in+medical+imaging%E2%80%93+shortcomings+and+recommendations%2E+arXiv+preprint+arXiv%3A2103%2E10292%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E10292%5D%28https%3A%2F%2Farxiv%2Eor)
109. Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020. [find](https://scholar.google.com/scholar?q=Nino+Vieillard%2C+Olivier+Pietquin%2C+and+Matthieu+Geist%2E+Munchausen+reinforcement+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%2C+2020%2E)
110. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Igor+Babuschkin%2C+Wojciech+M+Czarnecki%2C+Micha%C3%ABl+Mathieu%2C+Andrew+Dudzik%2C+Junyoung+Chung%2C+David+H+Choi%2C+Richard+Powell%2C+Timo+Ewalds%2C+Petko+Georgiev%2C+et+al%2E+Grandmaster+level+in+starcraft+i)
111. Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. arXiv preprint arXiv:2010.10814, 2020. — [arXiv:2010.10814](https://arxiv.org/abs/2010.10814) [find](https://scholar.google.com/scholar?q=Kaixin+Wang%2C+Bingyi+Kang%2C+Jie+Shao%2C+and+Jiashi+Feng%2E+Improving+generalization+in+reinforcement+learning+with+mixture+regularization%2E+arXiv+preprint+arXiv%3A2010%2E10814%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E10814%5D%28https%3A%2F)
112. Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar. Moving to a world beyond “p \\< 0.05”. The American Statistician, 2019. [find](https://scholar.google.com/scholar?q=Ronald+L%2E+Wasserstein%2C+Allen+L%2E+Schirm%2C+and+Nicole+A%2E+Lazar%2E+Moving+to+a+world+beyond+%E2%80%9Cp+%5C%3C+0%2E05%E2%80%9D%2E+The+American+Statistician%2C+2019%2E)
113. Bernard L Welch. The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika, 34(1/2):28–35, 1947. [find](https://scholar.google.com/scholar?q=Bernard+L+Welch%2E+The+generalization+ofstudent%E2%80%99s%E2%80%99+problem+when+several+different+population+variances+are+involved%2E+Biometrika%2C+34%281%2F2%29%3A28%E2%80%9335%2C+1947%2E)
114. Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019. — [arXiv:1910.01741](https://arxiv.org/abs/1910.01741) [find](https://scholar.google.com/scholar?q=Denis+Yarats%2C+Amy+Zhang%2C+Ilya+Kostrikov%2C+Brandon+Amos%2C+Joelle+Pineau%2C+and+Rob+Fergus%2E+Improving+sample+efficiency+in+model%2Dfree+reinforcement+learning+from+images%2E+arXiv+preprint+arXiv%3A1910%2E01741%2C+201)
115. Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. arXiv preprint arXiv:2010.07470, 2020. — [arXiv:2010.07470](https://arxiv.org/abs/2010.07470) [find](https://scholar.google.com/scholar?q=Jinhua+Zhu%2C+Yingce+Xia%2C+Lijun+Wu%2C+Jiajun+Deng%2C+Wengang+Zhou%2C+Tao+Qin%2C+and+Houqiang+Li%2E+Masked+contrastive+representation+learning+for+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2010%2E07470%2C+2020%2E+%E2%80%94+%5B)
116. Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. arXiv preprint arXiv:2106.11872, 2021. [find](https://scholar.google.com/scholar?q=Donglin+Zhuang%2C+Xingyao+Zhang%2C+Shuaiwen+Leon+Song%2C+and+Sara+Hooker%2E+Randomness+in+neural+network+training%3A+Characterizing+the+impact+of+tooling%2E+arXiv+preprint+arXiv%3A2106%2E11872%2C+2021%2E)

*Updated 2 November 2026*