---
title: "CORE-Bench — paper by Siegel et al. (2024)"
description: "The authors ask whether AI agents can take over this verification work, and they build CORE-Bench… 54 references, cited by 35 papers here."
authors: ["Zachary S. Siegel", "Sayash Kapoor", "Nitya Nadgir", "Benedikt Stroebl", "Arvind Narayanan"]
year: 2024
venue: "arXiv:2409.11363"
source: "https://arxiv.org/abs/2409.11363"
doi: "10.48550/arxiv.2409.11363"
arxiv: "2409.11363"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/siegel-2024-core-bench-fostering-credibility/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/siegel-2024-core-bench-fostering-credibility/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

# CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

**Authors:** [Zachary S. Siegel](https://md.research.elamur.ai/free/en/authors/zachary-s-siegel/), [Sayash Kapoor](https://md.research.elamur.ai/free/en/authors/sayash-kapoor/), [Nitya Nadgir](https://md.research.elamur.ai/free/en/authors/nitya-nadgir/), [Benedikt Stroebl](https://md.research.elamur.ai/free/en/authors/benedikt-stroebl/) and [Arvind Narayanan](https://md.research.elamur.ai/free/en/authors/arvind-narayanan/) · **Year:** 2024 · **Venue:** arXiv:2409.11363 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.11363) · [PDF](https://arxiv.org/pdf/2409.11363) · [DOI 10.48550/arxiv.2409.11363](https://doi.org/10.48550/arxiv.2409.11363) · [arXiv 2409.11363](https://arxiv.org/abs/2409.11363) · [OpenAlex](https://openalex.org/W4403707291)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/siegel-2024-core-bench-fostering-credibility/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many published studies cannot be rerun to obtain their reported numbers, even when the authors share code and data, and checking this by hand takes expert effort. The authors ask whether AI agents can take over this verification work, and they build CORE-Bench to measure it. Starting from 5,090 research capsules on CodeOcean, they kept 90 papers from computer science, social science, and medicine whose Python or R code they confirmed runs locally. They derived three tasks per paper, for 270 in total. Difficulty depends on how much help the agent gets. Easy supplies pre-computed outputs, Medium supplies a Dockerfile, and Hard supplies only the README, so the agent must install dependencies itself. Questions require reading either text output or figures. A parallel evaluation harness runs each task in its own VM and cuts evaluation time from weeks to hours. The baselines are AutoGPT and an adapted CORE-Agent, each run on GPT-4o and GPT-4o-mini. Light task-specific tuning helps substantially, yet the best agent solves only 21% of Hard tasks. The benchmark gives research-assistant agents a realistic target that can be updated over time.

## Key points
- **Dataset:** CORE-Bench has 270 tasks from 90 CodeOcean papers in three disciplines, with Python and R codebases. It is one of the few agent benchmarks that includes R. A task counts as solved only if every question about the repository is answered correctly.
- **Best results:** CORE-Agent with GPT-4o reached about 60% on CORE-Bench-Easy and 21% on CORE-Bench-Hard. AutoGPT, the general-purpose agent, scored notably lower.
- **Text vs. figures:** Questions based on text output were much easier than figure-based ones. On Easy, CORE-Agent with GPT-4o answered 87.88% of written questions and 59.26% of vision questions correctly. Python tasks were clearly easier than R tasks.
- **Failure modes:**
  - Agents often picked the wrong output file or figure.
  - They got stuck reinstalling dependencies until they hit the cost limit, and raising the limit did not help much.
  - Average cost was $0.54 for successful runs versus $2.59 for failed ones (CORE-Agent with GPT-4o).
- **Motivation:** In the 2022 ML reproducibility challenge, only 18 of 28 papers with code and data were fully reproducible. Code and harness are available at github.com/siegelz/core-bench.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 137 works · 53,849 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 96 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl and Arvind Narayanan. 2024. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv:2409.11363.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{siegel2024core,
  title = {{CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark}},
  author = {Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl and Arvind Narayanan},
  year = {2024},
  doi = {10.48550/arxiv.2409.11363},
  eprint = {2409.11363},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.11363}
}
```

- [← previous Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://md.research.elamur.ai/free/en/papers/tan-2024-personalized-pieces-efficient-personalized-large/)
- [next → Dual-Space Knowledge Distillation for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2024-dual-space-knowledge-distillation-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

54 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AISI. Inspect, 2024. URL [https://inspect.ai-safety-institute.org.uk](https://inspect.ai-safety-institute.org.uk). [↗](https://inspect.ai-safety-institute.org.uk)
2. Rose L. Andrew, Arianne Y.K. Albert, Sebastien Renaut, Diana J. Rennison, Dan G. Bock, and Tim Vines. Assessing the reproducibility of discriminant function analyses. PeerJ, 3:e1137, August 2015. ISSN 2167-8359. doi: 10.7717/peerj.1137. URL [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4540019/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4540019/). — [doi:10.7717/peerj.1137](https://doi.org/10.7717/peerj.1137) [↗](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4540019/)
3. Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. A Systematic Review of Reproducibility Research in Natural Language Processing. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational… [find](https://scholar.google.com/scholar?q=Anya+Belz%2C+Shubham+Agarwal%2C+Anastasia+Shimorina%2C+and+Ehud+Reiter%2E+A+Systematic+Review+of+Reproducibility+Research+in+Natural+Language+Processing%2E+In+Paola+Merlo%2C+Jorg+Tiedemann%2C+and+Reut+Tsarfaty+%28eds)
4. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Lintang+Sutawika%2C+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Alham+Fikri+Aji%2C+Pawan+Sasanka+Ammanamanchi%2C+Sidney+Black%2C+Jordan+Clive%2C+Anthony+DiPofi%2C+Julen+Etxaniz%2C+Benja)
5. Matteo Brivio and Çağrı Çöltekin. [Re] Exploring the Representation of Word Meanings in Context. ReScience C, 9(2):\#5, July 2023. doi: 10.5281/zenodo.8173658. URL [https://zenodo.org/records/8173658](https://zenodo.org/records/8173658). — [doi:10.5281/zenodo.8173658](https://doi.org/10.5281/zenodo.8173658) [↗](https://zenodo.org/records/8173658)
6. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/), July 2024. URL http://arxiv.org/abs/2407.21787. arXiv:2407.21787 [cs]. [↗](http://arxiv.org/abs/2407.21787)
7. Jonathan B. Buckheit and David L. Donoho. WaveLab and Reproducible Research. In Anestis Antoniadis and Georges Oppenheim (eds.), Wavelets and Statistics, pp. 55–81. Springer, New York, NY, 1995. ISBN 978-1-4612-2544-7. doi: 10.1007/978-1-4612-2544-7_5. URL https://doi.org/10.1007/978-1-4612-2544-7_… — [doi:10.1007/978-1-4612-2544-7_5](https://doi.org/10.1007/978-1-4612-2544-7_5) [↗](https://doi.org/10.1007/978-1-4612-2544-7_…)
8. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://md.research.elamur.ai/free/en/papers/cassano-2022-multipl-e-scalable-extensible-approach/),…
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
10. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Alijubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified, August 2024. URL [https://openai.com/index/](https://openai.com/index/) introducing-swe-benc… [↗](https://openai.com/index/)
11. April Clyburne-Sherin, Xu Fei, and Seth Ariel Green. Computational Reproducibility via Containers in Psychology. Meta-Psychology, 3, November 2019. ISSN 2003-2714. doi: 10.15626/MP.2018.892. URL [https://open.lnu.se/index.php/metapsychology/article/view/892](https://open.lnu.se/index.php/metapsychology/article/view/892). — [doi:10.15626/mp.2018.892](https://doi.org/10.15626/mp.2018.892) [↗](https://open.lnu.se/index.php/metapsychology/article/view/892)
12. Christian Collberg and Todd A. Proebsting. Repeatability in computer systems research. Communications of the ACM, 59(3):62–69, February 2016. ISSN 0001-0782, 1557-7317. doi: 10.1145/2812803. URL [https://dl.acm.org/doi/10.1145/2812803](https://dl.acm.org/doi/10.1145/2812803). — [doi:10.1145/2812803](https://doi.org/10.1145/2812803) [↗](https://dl.acm.org/doi/10.1145/2812803)
13. Paul Gertler, Sebastian Galiani, and Mauricio Romero. How to make replication the norm. Nature, 554(7693): 417–419, February 2018. doi: 10.1038/d41586-018-02108-9. URL [https://www.nature.com/articles/](https://www.nature.com/articles/) d41586-018-02108-9. Bandiera_abtest: a Cg_type: Comment Publisher: Nature Publishing Group Subject_… — [doi:10.1038/d41586-018-02108-9](https://doi.org/10.1038/d41586-018-02108-9) [↗](https://www.nature.com/articles/)
14. Kimberly J. Gilbert, Rose L. Andrew, Dan G. Bock, Michelle T. Franklin, Nolan C. Kane, Jean-Sébastien Moore, Brook T. Moyers, Sébastien Renaut, Diana J. Rennison, Thor Veen, and Timothy H. Vines. Recommendations for utilizing and reporting population genetic analyses: the reproducibility of genetic… — [doi:10.1111/j.1365-294x.2012.05754.x](https://doi.org/10.1111/j.1365-294x.2012.05754.x) [find](https://scholar.google.com/scholar?q=Kimberly+J%2E+Gilbert%2C+Rose+L%2E+Andrew%2C+Dan+G%2E+Bock%2C+Michelle+T%2E+Franklin%2C+Nolan+C%2E+Kane%2C+Jean%2DS%C3%A9bastien+Moore%2C+Brook+T%2E+Moyers%2C+S%C3%A9bastien+Renaut%2C+Diana+J%2E+Rennison%2C+Thor+Veen%2C+and+Timothy+H%2E+Vines%2E+Reco)
15. Tom E. Hardwicke, Maya B. Mathur, Kyle MacDonald, Gustav Nilsonne, George C. Banks, Mallory C. Kidwell, Alicia Hofelich Mohr, Elizabeth Clayton, Erica J. Yoon, Michael Henry Tessler, Richie L. Lenne, Sara Altman, Bria Long, and Michael C. Frank. Data availability, reusability, and analytic reproduci… — [doi:10.1098/rsos.180448](https://doi.org/10.1098/rsos.180448) [find](https://scholar.google.com/scholar?q=Tom+E%2E+Hardwicke%2C+Maya+B%2E+Mathur%2C+Kyle+MacDonald%2C+Gustav+Nilsonne%2C+George+C%2E+Banks%2C+Mallory+C%2E+Kidwell%2C+Alicia+Hofelich+Mohr%2C+Elizabeth+Clayton%2C+Erica+J%2E+Yoon%2C+Michael+Henry+Tessler%2C+Richie+L%2E+Lenne%2C+)
16. Tom E. Hardwicke, Manuel Bohn, Kyle MacDonald, Emily Hembacher, Michèle B. Nuijten, Benjamin N. Peloquin, Benjamin E. deMayo, Bria Long, Erica J. Yoon, and Michael C. Frank. Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : an observational study.… — [doi:10.1098/rsos.201494](https://doi.org/10.1098/rsos.201494) [find](https://scholar.google.com/scholar?q=Tom+E%2E+Hardwicke%2C+Manuel+Bohn%2C+Kyle+MacDonald%2C+Emily+Hembacher%2C+Mich%C3%A8le+B%2E+Nuijten%2C+Benjamin+N%2E+Peloquin%2C+Benjamin+E%2E+deMayo%2C+Bria+Long%2C+Erica+J%2E+Yoon%2C+and+Michael+C%2E+Frank%2E+Analytic+reproducibility+i)
17. Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The Larger the Better? Improved LLM Code-Generation via Budget Reallocation, July 2024. URL http://arxiv.org/abs/2404.00725. arXiv:2404.00725 [cs]. — [arXiv:2404.00725](https://arxiv.org/abs/2404.00725) [↗](http://arxiv.org/abs/2404.00725)
18. Yifeng He, Ethan Wang, Yuyang Rong, Zifei Cheng, and Hao Chen. Security of AI Agents, 2024. URL https://arxiv.org/abs/2406.08689. Version Number: 2. — [arXiv:2406.08689](https://arxiv.org/abs/2406.08689) [↗](https://arxiv.org/abs/2406.08689)
19. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking Large Language Models As AI Research Agents, October 2023. [URL https://arxiv.org/abs/2310.03302v1](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). [↗](https://arxiv.org/abs/2310.03302v1)
20. John P. A. Ioannidis, David B. Allison, Catherine A. Ball, Issa Coulibaly, Xiangqin Cui, Aedín C. Culhane, Mario Falchi, Cesare Furlanello, Laurence Game, Giuseppe Jurman, Jon Mangion, Tapan Mehta, Michael Nitzberg, Grier P. Page, Enrico Petretto, and Vera van Noort. Repeatability of published micro… — [doi:10.1038/ng.295](https://doi.org/10.1038/ng.295) [find](https://scholar.google.com/scholar?q=John+P%2E+A%2E+Ioannidis%2C+David+B%2E+Allison%2C+Catherine+A%2E+Ball%2C+Issa+Coulibaly%2C+Xiangqin+Cui%2C+Aed%C3%ADn+C%2E+Culhane%2C+Mario+Falchi%2C+Cesare+Furlanello%2C+Laurence+Game%2C+Giuseppe+Jurman%2C+Jon+Mangion%2C+Tapan+Mehta%2C+Mi)
21. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/), October 2023. URL https://arxiv.org/abs/2310.06770v1. [↗](https://arxiv.org/abs/2310.06770v1)
22. Sayash Kapoor and Arvind Narayanan. Evaluating LLMs is a minefield, October 2023. URL [https://www.aisnakeoil.com/p/evaluating-llms-is-a-minefield](https://www.aisnakeoil.com/p/evaluating-llms-is-a-minefield). [↗](https://www.aisnakeoil.com/p/evaluating-llms-is-a-minefield)
23. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. [AI Agents That Matter](https://md.research.elamur.ai/free/en/papers/kapoor-2024-ai-agents-that-matter/), July 2024. URL http://arxiv.org/abs/2407.01502. arXiv:2407.01502 [cs]. [↗](http://arxiv.org/abs/2407.01502)
24. Markus Konkol, Christian Kray, and Max Pfeiffer. Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science, 33(2):408–429, February 2019. ISSN 1365-8816. doi: 10.1080… — [doi:10.1080/13658816](https://doi.org/10.1080/13658816) [find](https://scholar.google.com/scholar?q=Markus+Konkol%2C+Christian+Kray%2C+and+Max+Pfeiffer%2E+Computational+reproducibility+in+geoscientific+papers%3A+Insights+from+a+series+of+studies+with+geoscientists+and+a+reproduction+study%2E+International+Jou)
25. Jimmy Koppel. Everyone’s talking about Sakana’s AI scientist. But no-one’s answering the big question: is its output good?, August 2024. URL [https://x.com/jimmykoppel/status/1828077203956850756](https://x.com/jimmykoppel/status/1828077203956850756). [↗](https://x.com/jimmykoppel/status/1828077203956850756)
26. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
27. David M. Liu and Matthew J. Salganik. Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge. Socius, 5:2378023119849803, January 2019. ISSN 2378-0231. doi: 10.1177/2378023119849803. URL https://doi.org/10.1177/2378023119849803. Publisher: SAGE Public… — [doi:10.1177/2378023119849803](https://doi.org/10.1177/2378023119849803) [↗](https://doi.org/10.1177/2378023119849803)
28. Victor Livernoche and Vidya Sujaya. [Re] A Reproduction of Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification. ReScience C, 9(2):\#33, July 2023. doi: 10.5281/zenodo.8173735. URL [https://zenodo.org/records/8173735](https://zenodo.org/records/8173735). — [doi:10.5281/zenodo.8173735](https://doi.org/10.5281/zenodo.8173735) [↗](https://zenodo.org/records/8173735)
29. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/), August 2024. URL http://arxiv.org/abs/ 2408.06292. arXiv:2408.06292 [cs]. [↗](http://arxiv.org/abs/)
30. Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, F… [find](https://scholar.google.com/scholar?q=Arjun+Majumdar%2C+Anurag+Ajay%2C+Xiaohan+Zhang%2C+Pranav+Putta%2C+Sriram+Yenamandra%2C+Mikael+Henaff%2C+Sneha+Silwal%2C+Paul+Mcvay%2C+Oleksandr+Maksymets%2C+Sergio+Arnaud%2C+Karmesh+Yadav%2C+Qiyang+Li%2C+Ben+Newman%2C+Mohit+Sh)
31. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://md.research.elamur.ai/free/en/papers/majumder-2024-discoverybench-data-driven-discovery/), July 2024. URL http://arxiv.org/abs/2407.01725.… [↗](http://arxiv.org/abs/2407.01725.…)
32. B. D. McCullough, Kerry Anne McGeary, and Teresa D. Harrison. Lessons from the JMCB Archive. Journal of Money, Credit and Banking, 38(4):1093–1107, 2006. ISSN 0022-2879. URL [https://www.jstor.org/](https://www.jstor.org/) stable/3838995. Publisher: [Wiley, Ohio State University Press]. [↗](https://www.jstor.org/)
33. National Academies of Sciences Engineering and Medicine. Reproducibility and Replicability in Science. 2019. doi: 10.17226/25303. URL [https://nap.nationalacademies.org/read/25303/chapter/7](https://nap.nationalacademies.org/read/25303/chapter/7). — [doi:10.17226/25303](https://doi.org/10.17226/25303) [↗](https://nap.nationalacademies.org/read/25303/chapter/7)
34. METR. Vivaria, 2024. URL [https://vivaria.metr.org](https://vivaria.metr.org). [↗](https://vivaria.metr.org)
35. Florian Naudet, Charlotte Sakarovitch, Perrine Janiaud, Ioana Cristea, Daniele Fanelli, David Moher, and John P. A. Ioannidis. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS M… — [doi:10.1136/bmj.k400](https://doi.org/10.1136/bmj.k400) [find](https://scholar.google.com/scholar?q=Florian+Naudet%2C+Charlotte+Sakarovitch%2C+Perrine+Janiaud%2C+Ioana+Cristea%2C+Daniele+Fanelli%2C+David+Moher%2C+and+John+P%2E+A%2E+Ioannidis%2E+Data+sharing+and+reanalysis+of+randomized+controlled+trials+in+leading+bi)
36. Pepijn Obels, Daniël Lakens, Nicholas A. Coles, Jaroslav Gottfried, and Seth A. Green. Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology. Advances in Methods and Practices in Psychological Science, 3(2):229–237, June 2020. ISSN 2515-2459. doi: 10.1177/251524… — [doi:10.1177/2515245920918872](https://doi.org/10.1177/2515245920918872) [find](https://scholar.google.com/scholar?q=Pepijn+Obels%2C+Dani%C3%ABl+Lakens%2C+Nicholas+A%2E+Coles%2C+Jaroslav+Gottfried%2C+and+Seth+A%2E+Green%2E+Analysis+of+Open+Data+and+Computational+Reproducibility+in+Registered+Reports+in+Psychology%2E+Advances+in+Methods+)
37. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d’Alche Buc, Emily Fox, and Hugo Larochelle. Improving Reproducibility in Machine Learning Research(A Report from the NeurIPS 2019 Reproducibility Program). Journal of Machine Learning Research, 22… [find](https://scholar.google.com/scholar?q=Joelle+Pineau%2C+Philippe+Vincent%2DLamarre%2C+Koustuv+Sinha%2C+Vincent+Lariviere%2C+Alina+Beygelzimer%2C+Florence+d%E2%80%99Alche+Buc%2C+Emily+Fox%2C+and+Hugo+Larochelle%2E+Improving+Reproducibility+in+Machine+Learning+Resear)
38. Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. [CiteME: Can Language Models Accurately Cite Scientific Claims?](https://md.research.elamur.ai/free/en/papers/press-2024-citeme-language-models-accurately/), July 2024. URL [http://arxiv.org/](http://arxiv.org/) abs/2407.12861. arXiv:2407.12861 [cs]. [↗](http://arxiv.org/)
39. Christophe Pérignon, Olivier Akmansoy, Christophe Hurlin, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Albert J. Menkveld, Michael Razen, and Utz Weitzel. Computational Reproducibility in Finance: Evidence from 1,000 Tests, February 2024. URL [https://papers.s…](https://papers.s…) [↗](https://papers.s…)
40. Edward Raff. A Step Toward Quantifying Independently Reproducible Machine Learning Research, September 2019. URL http://arxiv.org/abs/1909.06674. arXiv:1909.06674 [cs, stat]. [↗](http://arxiv.org/abs/1909.06674)
41. Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the Everything in the Whole Wide World Benchmark, 2021. URL https://arxiv.org/abs/2111.15366. Version Number: 1. [↗](https://arxiv.org/abs/2111.15366)
42. Sheeba Samuel and Daniel Mietchen. Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience, 13:giad113, January 2024. ISSN 2047-217X. doi: 10.1093/gigascience/giad113. URL https://doi.org/10.1093/gigascience/giad113. — [doi:10.1093/gigascience/giad113](https://doi.org/10.1093/gigascience/giad113) [↗](https://doi.org/10.1093/gigascience/giad113)
43. Significant Gravitas. AutoGPT, September 2024. URL [https://github.com/Significant-Gravitas/](https://github.com/Significant-Gravitas/) AutoGPT. original-date: 2023-03-16T09:21:07Z. [↗](https://github.com/Significant-Gravitas/)
44. Koustuv Sinha, Maurits Bleeker, Samarth Bhargav, Jessica Zosa Forde, Sharath Chandra Raparthy, Jesse Dodge, Joelle Pineau, and Robert Stojnic. ML Reproducibility Challenge 2022. ReScience C, 9(2):\#46, July 2023. doi: 10.5281/zenodo.8200058. URL [https://zenodo.org/records/8200058](https://zenodo.org/records/8200058). — [doi:10.5281/zenodo.8200058](https://doi.org/10.5281/zenodo.8200058) [↗](https://zenodo.org/records/8200058)
45. Jeffrey R. Spence and David J. Stanley. Prediction Interval: What to Expect When You’re Expecting . . . A Replication. PLOS ONE, 11(9):e0162874, September 2016. ISSN 1932-6203. doi: 10.1371/journal.pone. 0162874. URL [https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0162874](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0162874). Publishe… — [doi:10.1371/journal.pone](https://doi.org/10.1371/journal.pone) [↗](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0162874)
46. Daniel Stockemer, Sebastian Koehler, and Tobias Lentz. Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature. PS: Political Science & Politics, 51 (4):799–803, October 2018. ISSN 1049-0965, 1537-5935. doi: 10.1017/S1049096518000926. URL [https://www.cambridge…](https://www.cambridge…) — [doi:10.1017/s1049096518000926](https://doi.org/10.1017/s1049096518000926) [↗](https://www.cambridge…)
47. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… — [SciCode: A Research Coding Benchmark Curated by Scientists](https://md.research.elamur.ai/free/en/papers/tian-2024-scicode-research-coding-benchmark/)
48. Ana Trisovic, Matthew K. Lau, Thomas Pasquier, and Mercè Crosas. A large-scale study on research code quality and execution. Scientific Data, 9(1):60, February 2022. ISSN 2052-4463. doi: 10.1038/ s41597-022-01143-6. URL [https://www.nature.com/articles/s41597-022-01143-6](https://www.nature.com/articles/s41597-022-01143-6). Publisher: Nature Publishing… [↗](https://www.nature.com/articles/s41597-022-01143-6)
49. Benjamin D. K. Wood, Rui Müller, and Annette N. Brown. Push button replication: Is impact evaluation evidence for international development verifiable? PLOS ONE, 13(12):e0209416, December 2018. ISSN 1932-6203. doi: 10.1371/journal.pone.0209416. URL [https://journals.plos.org/plosone/article?](https://journals.plos.org/plosone/article?) id=10.13… — [doi:10.1371/journal.pone.0209416](https://doi.org/10.1371/journal.pone.0209416) [↗](https://journals.plos.org/plosone/article?)
50. Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. ChartBench: A Benchmark for Complex Visual Reasoning in Charts, June 2024. URL http://arxiv.org/abs/2312.15915. arXiv:2312.15915 [cs]. — [arXiv:2312.15915](https://arxiv.org/abs/2312.15915) [↗](http://arxiv.org/abs/2312.15915)
51. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. [SWE-AGENT: Agent-Computer Interfaces Enable Automated Software Engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/). 2024. URL [https://swe-agent.com/paper.pdf](https://swe-agent.com/paper.pdf). [↗](https://swe-agent.com/paper.pdf)
52. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024. [URL http://arxiv.org/abs/2406.12045](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/). arXiv:2406.12045 [cs]. [↗](http://arxiv.org/abs/2406.12045)
53. Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AI Systems, February 2024. URL [http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/](http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/). [↗](http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/)
54. Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai. PyBench: Evaluating LLM Agent on various real-world coding tasks, August 2024. URL http://arxiv.org/abs/2407.16732. arXiv:2407.16732 [cs]. — [arXiv:2407.16732](https://arxiv.org/abs/2407.16732) [↗](http://arxiv.org/abs/2407.16732)

*Updated 3 November 2026*