--- title: "Reproducibility in Machine Learning-Driven Research (2023)" description: "The authors review existing literature with three goals: describe the state of ML… 86.6% accuracy. 12 references, cited by 3 papers here." authors: ["Harald Semmelrock", "Simone Kopeinik", "Dieter Theiler", "Tony Ross-Hellauer", "Dominik Kowald"] year: 2023 venue: "arXiv:2307.10320" source: "https://arxiv.org/abs/2307.10320" doi: "10.48550/arxiv.2307.10320" arxiv: "2307.10320" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/semmelrock-2023-reproducibility-machine-learning-driven/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/semmelrock-2023-reproducibility-machine-learning-driven/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/15/) / Reproducibility in Machine Learning-Driven Research # Reproducibility in Machine Learning-Driven Research **Authors:** [Harald Semmelrock](https://md.research.elamur.ai/free/en/authors/harald-semmelrock/), [Simone Kopeinik](https://md.research.elamur.ai/free/en/authors/simone-kopeinik/), [Dieter Theiler](https://md.research.elamur.ai/free/en/authors/dieter-theiler/), [Tony Ross-Hellauer](https://md.research.elamur.ai/free/en/authors/tony-ross-hellauer/) and [Dominik Kowald](https://md.research.elamur.ai/free/en/authors/dominik-kowald/) · **Year:** 2023 · **Venue:** arXiv:2307.10320 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2307.10320) · [PDF](https://arxiv.org/pdf/2307.10320) · [DOI 10.48550/arxiv.2307.10320](https://doi.org/10.48550/arxiv.2307.10320) · [arXiv 2307.10320](https://arxiv.org/abs/2307.10320) · [OpenAlex](https://openalex.org/W4385018165) **Citations:** 8 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/semmelrock-2023-reproducibility-machine-learning-driven/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This mini survey examines why so many machine learning (ML) results cannot be reproduced, and what might help. The authors review existing literature with three goals: describe the state of ML reproducibility across disciplines, catalogue the barriers, and collect the drivers (tools, practices, interventions) that support it. They adopt a three-level framework: experiment reproducibility (R1), data reproducibility (R2) and method reproducibility (R3). Each step up trades exact repeatability for broader generalizability. The review covers computer science (deep learning, reinforcement learning, NLP, recommender systems) and health/life science. Common barriers include unshared code and data, nondeterministic training, sensitivity to library versions, and data leakage. Medical data adds privacy constraints. The surveyed remedies include containerized environments, reproducibility checklists, model info sheets, awareness initiatives and stricter journal policies. The paper offers no new experiments. Its value lies in a shared vocabulary and an organized map of barriers and drivers, which can help researchers choose suitable reproducibility measures. ## Key points - **Three degrees of reproducibility:** - R1 requires the same code and the same data. - R2 requires a different implementation on the same data. - R3 requires a different implementation on different data. - The authors relate these levels to the ACM's distinction between reproducibility and replicability. - **ML platforms:** Running one experiment on different platforms (e.g., OpenML, Kaggle, Azure ML) can produce different outcomes. - **NLP findings:** - Simply rerunning code can change results. - Minor settings such as rare-word thresholds can shift performance strongly. - When a reproduction deviated from the original, it scored lower. - **Checklists:** A reproducibility checklist proved useful in the NeurIPS 2019 challenge. GPT-4 checked whether checklist items were fulfilled with 86.6% accuracy. - **Other drivers:** - Docker and Code Ocean for standardized environments. - Model info sheets to detect data leakage. - ReScience and ReproducedPapers.org to raise awareness. - Preregistration and reproducibility papers in ACM TORS. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Graz University of Technology](https://md.research.elamur.ai/free/en/universities/graz-university-of-technology/) — University, Graz, Austria · 5 works · 108 citations - [Know Center Research GmbH](https://md.research.elamur.ai/free/en/companies/know-center-research-gmbh/) — Company, Graz, Austria · 3 works · 104 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 96 works ## Cite Cite as: Harald Semmelrock, Simone Kopeinik, Dieter Theiler, Tony Ross-Hellauer and Dominik Kowald. 2023. Reproducibility in Machine Learning-Driven Research. arXiv:2307.10320. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{semmelrock2023reproducibility, title = {{Reproducibility in Machine Learning-Driven Research}}, author = {Harald Semmelrock, Simone Kopeinik, Dieter Theiler, Tony Ross-Hellauer and Dominik Kowald}, year = {2023}, doi = {10.48550/arxiv.2307.10320}, eprint = {2307.10320}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2307.10320} } ``` - [← previous Optimizing the Size of Subword Vocabularies in Dialect Classification](https://md.research.elamur.ai/free/en/papers/kanjirangat-2023-optimizing-size-subword-vocabularies-dialect/) - [next → TCE at Qur'an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur'anic QA](https://md.research.elamur.ai/free/en/papers/elkomy-2023-tce-qur-qa-2023-shared/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/15/) ## References 12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. Abel, D.: simple rl: Reproducible Reinforcement Learning in Python. ICLR (2019) 2. Ahmed, H., Lofstead, J.: Managing Randomness to Enable Reproducible Machine Learning. In: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems. pp. 15–20. [find](https://scholar.google.com/scholar?q=1%2E+Abel%2C+D%2E%3A+simple+rl%3A+Reproducible+Reinforcement+Learning+in+Python%2E+ICLR+%282019%29+2%2E+Ahmed%2C+H%2E%2C+Lofstead%2C+J%2E%3A+Managing+Randomness+to+Enable+Reproducible+Machine+Learning%2E+In%3A+Proceedings+of+the+5th+I) 2. ACM, Minneapolis MN USA (2022). https://doi.org/10.1145/3526062.3536353, https://dl.acm.org/doi/ 10.1145/3526062.3536353 3. Artrith, N., Butler, K.T., Coudert, F.X., Han, S., Isayev, O., Jain, A., Walsh, A.: Best practices in machine learning for chemistry. Nature Chemistry 13(6), 505–508 (2021). ht… [↗](https://doi.org/10.1145/3526062.3536353) 3. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym (2016). https://doi.org/10.48550/arXiv.1606. 01540, http://arxiv.org/abs/1606.01540, arXiv:1606.01540 [cs] 11. Clyburne-Sherin, A., Fei, X., Green, S.A.: Computational Reproducibility via Contain… [↗](https://doi.org/10.48550/arXiv.1606) 4. Reproducibility in Machine Learning-Driven Research 14. Dror, R., Baumer, G., Bogomolov, M., Reichart, R.: [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://md.research.elamur.ai/free/en/papers/dror-2017-replicability-analysis-natural-language-processing/). Transactions of the Association for Computational Linguistics 5, 471–486 (2017) 15. Ferigo,… 5. Science (2018). https://doi.org/10.1126/science.aat3298 — [doi:10.1126/science.aat3298](https://doi.org/10.1126/science.aat3298) [↗](https://doi.org/10.1126/science.aat3298) 6. Semmelrock, Kopeinik, Theiler, Ross-Hellauer, and Kowald 26. Kapoor, S., Narayanan, A.: Leakage and the Reproducibility Crisis in ML-based Science (2022), http://arxiv.org/abs/2207.07048, arXiv:2207.07048 [cs, stat] 27. Khetarpal, K., Ahmed, Z., Cianflone, A., Islam, R., Pineau, J.: RE-EVALUATE: Rep… [↗](http://arxiv.org/abs/2207.07048) 7. ICML (2018), [https://openreview.net/forum?id=HJgAmITcgm](https://openreview.net/forum?id=HJgAmITcgm) 28. Kowald, D., Schedl, M., Lex, E.: The unfairness of popularity bias in music recommendation: A reproducibility study. In: Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2… [↗](https://openreview.net/forum?id=HJgAmITcgm) 8. Springer (2020) 29. Liu, R., Shah, N.B.: [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://md.research.elamur.ai/free/en/papers/liu-2023-reviewergpt-exploratory-study-using/) (2023). https://doi.org/10.48550/arXiv. 2306.00622, http://arxiv.org/abs/2306.00622, arXiv:2306.00622 [cs] 30. McDermott, M.B.A., Wang, S., Marinsek, N., Ranganath, R., Ghass… [↗](https://doi.org/10.48550/arXiv) 9. Springer (2021) 32. Nagarajan, P., Warnell, G., Stone, P.: The Impact of Nondeterminism on Reproducibility in Deep Reinforcement Learning. [find](https://scholar.google.com/scholar?q=Springer+%282021%29+32%2E+Nagarajan%2C+P%2E%2C+Warnell%2C+G%2E%2C+Stone%2C+P%2E%3A+The+Impact+of+Nondeterminism+on+Reproducibility+in+Deep+Reinforcement+Learning%2E) 10. ICML (2018), [https://openreview.net/forum?id=S1e-OsZ4e7](https://openreview.net/forum?id=S1e-OsZ4e7) 33. Nagarajan, P., Warnell, G., Stone, P.: Deterministic Implementations for Reproducibility in Deep Reinforcement Learning (2019). https://doi.org/10.48550/ arXiv.1809.05676, http://arxiv.org/abs/1809.05676, arXiv:1809.05676 [cs] 34. Nalepa,… [↗](https://openreview.net/forum?id=S1e-OsZ4e7) 11. Reproducibility in Machine Learning-Driven Research 39. Pouchard, L., Lin, Y., Van Dam, H.: Replicating Machine Learning Experiments in Materials Science. In: Parallel Computing: Technology Trends, pp. 743–755. IOS Press (2020). https://doi.org/10.3233/APC200105, https://ebooks.iospress. nl/doi/10.3… — [doi:10.3233/apc200105](https://doi.org/10.3233/apc200105) [↗](https://doi.org/10.3233/APC200105) 12. College & Research Libraries (2018). https://doi.org/https://doi.org/10.5860/crl.79.1.2, https://journals.acrl.org/index.php/crl/article/view/16846 42. Shahriari, M., Ramler, R., Fischer, L.: How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models? Machine Learni… — [doi:10.5860/crl.79.1.2](https://doi.org/10.5860/crl.79.1.2) [find](https://scholar.google.com/scholar?q=College+%26+Research+Libraries+%282018%29%2E+https%3A%2F%2Fdoi%2Eorg%2Fhttps%3A%2F%2Fdoi%2Eorg%2F10%2E5860%2Fcrl%2E79%2E1%2E2%2C+https%3A%2F%2Fjournals%2Eacrl%2Eorg%2Findex%2Ephp%2Fcrl%2Farticle%2Fview%2F16846+42%2E+Shahriari%2C+M%2E%2C+Ramler%2C+R%2E%2C+Fischer%2C+L%2E%3A+How+Do+D) *Updated 3 November 2026*