---
title: "Paper2Code — paper by Seo et al. (2025) — summary and ideas"
description: "The authors ask whether an LLM system can build a working repository from the paper alone, without… 68 references, cited by 24 papers here."
authors: ["Minju Seo", "Jinheon Baek", "Seongyun Lee", "Sung Ju Hwang"]
year: 2025
venue: "arXiv:2504.17192"
source: "https://arxiv.org/abs/2504.17192"
doi: "10.48550/arxiv.2504.17192"
arxiv: "2504.17192"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/seo-2025-paper2code-automating-code-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/seo-2025-paper2code-automating-code-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

# Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

**Authors:** Minju Seo, [Jinheon Baek](https://md.research.elamur.ai/free/en/authors/jinheon-baek/), [Seongyun Lee](https://md.research.elamur.ai/free/en/authors/seongyun-lee/) and [Sung Ju Hwang](https://md.research.elamur.ai/free/en/authors/sung-ju-hwang/) · **Year:** 2025 · **Venue:** arXiv:2504.17192 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.17192) · [PDF](https://arxiv.org/pdf/2504.17192) · [DOI 10.48550/arxiv.2504.17192](https://doi.org/10.48550/arxiv.2504.17192) · [arXiv 2504.17192](https://arxiv.org/abs/2504.17192) · [OpenAlex](https://openalex.org/W4416872478)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [24](https://md.research.elamur.ai/free/en/papers/seo-2025-paper2code-automating-code-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many machine learning papers are published without code. Across ICLR, ICML and NeurIPS 2024, only about one in five accepted papers released an official implementation, so reproducing results often requires slow reverse-engineering. The authors ask whether an LLM system can build a working repository from the paper alone, without existing code, APIs or supplementary material.

Their answer is PaperCoder, a multi-agent framework that mimics how developers approach a project:
- **Planning:** it drafts a roadmap, produces class and sequence diagrams, orders the files by dependency and writes configuration files.
- **Analysis:** it works out the details of each file, such as inputs, outputs, interactions between modules and constraints taken from the paper.
- **Generation:** it writes the code file by file, following the established order.

The system is tested on a new benchmark, Paper2CodeBench, and on PaperBench. Evaluation combines model-based judging (with and without author repositories as reference) and ratings from the original paper authors. PaperCoder clearly beats the baselines, and its output usually runs with little repair. This makes it a practical aid for reproducing research.

## Key points
- On average only 19.5% of papers accepted at the three major ML venues in 2024 shared official code. This gap motivated the work.
- Paper2CodeBench is built from recent ICLR, ICML and NeurIPS papers. PaperCoder also outperforms strong baselines on PaperBench (Starace et al., 2025).
- In the human evaluation by paper authors, 88% judged PaperCoder's repository the best among the alternatives, and 92% found it helpful for reproduction.
- When generated code failed to run, fixing it took small edits to about 0.81% of code lines on average. Ablations indicate that each of the three stages adds to performance.
- Code is released at github.com/going-doer/Paper2Code. Reference-free model judging used gpt-4o-2024-11-20.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 96 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 194 works

## Cite

Cite as: Minju Seo, Jinheon Baek, Seongyun Lee and Sung Ju Hwang. 2025. Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning. arXiv:2504.17192.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{seo2025paper2code,
  title = {{Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning}},
  author = {Minju Seo, Jinheon Baek, Seongyun Lee and Sung Ju Hwang},
  year = {2025},
  doi = {10.48550/arxiv.2504.17192},
  eprint = {2504.17192},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.17192}
}
```

- [← previous OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain](https://md.research.elamur.ai/free/en/papers/wang-2025-omnieval-omnidirectional-automatic-rag-evaluation/)
- [next → Personalized Generation In Large Model Era: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2025-personalized-generation-large-model-era/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

68 references, 45 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/), 2021. URL https://arxiv.org/abs/2108.07732. [↗](https://arxiv.org/abs/2108.07732)
2. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. [Researchagent: Iterative research idea generation over scientific literature with large language models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/), 2025. URL https://arxiv.org/abs/2404.07738. [↗](https://arxiv.org/abs/2404.07738)
3. Monya Baker. 1,500 scientists lift the lid on reproducibility, 2016. URL [https://www.nature.com/articles/533452a](https://www.nature.com/articles/533452a). [↗](https://www.nature.com/articles/533452a)
4. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M adry. [Mle-bench: Evaluating machine learning agents on machine learning engineering](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/), 2025. URL https://arxiv.org/abs/2410.0709… [↗](https://arxiv.org/abs/2410.0709…)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349 (6251):aac4716, 2015. doi: 10.1126/science.aac4716. URL [https://www.science.org/](https://www.science.org/) doi/abs/10.1126/science.aac4716. — [doi:10.1126/science.aac4716](https://doi.org/10.1126/science.aac4716) [↗](https://www.science.org/)
7. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. [Marg: Multi-agent review generation for scientific papers](https://md.research.elamur.ai/free/en/papers/arcy-2024-marg-multi-agent-review/), 2024. URL https://arxiv.org/abs/2401.04259. [↗](https://arxiv.org/abs/2401.04259)
8. DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie,… — [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2024-deepseek-coder-v2-breaking-barrier/)
9. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
11. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo… [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+Gptscore%3A+Evaluate+as+you+desire%2E+In+Kevin+Duh%2C+Helena+G%C3%B3mez%2DAdorno%2C+and+Steven+Bethard+%28eds%2E%29%2C+Proceedings+of+the+2024+Conference+of+the+Nort)
12. Kanishk Gandhi, Ayush K Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. [Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective STars](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/). In Second Conference on Language Modeling, 2025. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) QGJ9ttXLTy. [↗](https://openreview.net/forum?id=)
13. Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. Opendatalab: Empowering general artificial intelligence with open datasets. arXiv preprint arXiv:2407.13773, 2024. — [arXiv:2407.13773](https://arxiv.org/abs/2407.13773) [find](https://scholar.google.com/scholar?q=Conghui+He%2C+Wei+Li%2C+Zhenjiang+Jin%2C+Chao+Xu%2C+Bin+Wang%2C+and+Dahua+Lin%2E+Opendatalab%3A+Empowering+general+artificial+intelligence+with+open+datasets%2E+arXiv+preprint+arXiv%3A2407%2E13773%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E13)
14. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. [Measuring coding challenge competence with APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Proces…
15. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. [Metagpt: Meta programming for A multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). In The Twelfth…
16. OpenReview.net, 2024. URL [https://openreview.net/forum?id=VtmBAGCN7o](https://openreview.net/forum?id=VtmBAGCN7o). [↗](https://openreview.net/forum?id=VtmBAGCN7o)
17. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. [Mlagentbench: Evaluating language agents on machine learning experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
18. OpenReview.net, 2024. URL [https://openreview.net/forum?id=1Fs1LvjYQW](https://openreview.net/forum?id=1Fs1LvjYQW). [↗](https://openreview.net/forum?id=1Fs1LvjYQW)
19. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen… — [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
20. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/), 2024. URL https://arxiv.org/abs/2403. 07974. [↗](https://arxiv.org/abs/2403)
21. Steven A. Lehr, Aylin Caliskan, Suneragiri Liyanage, and Mahzarin R. Banaji. Chatgpt as research scientist: Probing gpt’s capabilities as a research librarian, research ethicist, data generator, and data predictor. Proceedings of the National Academy of Sciences, 121(35):e2404328121, 2024. doi: 10.1… — [doi:10.1073/pnas.2404328121](https://doi.org/10.1073/pnas.2404328121) [find](https://scholar.google.com/scholar?q=Steven+A%2E+Lehr%2C+Aylin+Caliskan%2C+Suneragiri+Liyanage%2C+and+Mahzarin+R%2E+Banaji%2E+Chatgpt+as+research+scientist%3A+Probing+gpt%E2%80%99s+capabilities+as+a+research+librarian%2C+research+ethicist%2C+data+generator%2C+and+d)
22. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. [Chain of ideas: Revolutionizing research via novel idea development with llm agents](https://md.research.elamur.ai/free/en/papers/li-2024-chain-ideas-revolutionizing-research/), 2024. URL https://arxiv.org/abs/2410.1318… [↗](https://arxiv.org/abs/2410.1318…)
23. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
24. Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiarui Zhang, Xinyu Wang, and Xiang Bai. Monkeyocr: Document parsing with a structure-recognition-relation triplet paradigm, 2025. URL https://arxiv.org/abs/2506.05218. [↗](https://arxiv.org/abs/2506.05218)
25. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. [Can large language models provide useful feedback on research papers? a large-scale empirical analysis](https://md.research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/). NEJM AI, 1(8):AIoa240…
26. Tianyang Liu, Canwen Xu, and Julian J. McAuley. [Repobench: Benchmarking repository-level code auto-completion systems](https://md.research.elamur.ai/free/en/papers/liu-2023-repobench-benchmarking-repository-level-code/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
27. OpenReview.net, 2024. URL [https://openreview.net/forum?id=pPjZIOuQuF](https://openreview.net/forum?id=pPjZIOuQuF). [↗](https://openreview.net/forum?id=pPjZIOuQuF)
28. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. [G-eval: NLG evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa…
29. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. [S2ORC: The semantic scholar open research corpus](https://md.research.elamur.ai/free/en/papers/lo-2019-s2orc-semantic-scholar-open-research/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983, Online, July 2020. Association for Computational Linguistics. doi: 10.1865…
30. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/), 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2408.06292. [↗](https://arxiv.org/)
31. Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. Repoagent: An llm-powered open-source framework for repository-level code documentation generation, 2024. URL https://arxiv.org/abs/2402.16667… [↗](https://arxiv.org/abs/2402.16667…)
32. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
33. Ian Magnusson, Noah A. Smith, and Jesse Dodge. Reproducibility in NLP: what have we learned from the checklist? In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 12789–12811. As… [find](https://scholar.google.com/scholar?q=Ian+Magnusson%2C+Noah+A%2E+Smith%2C+and+Jesse+Dodge%2E+Reproducibility+in+NLP%3A+what+have+we+learned+from+the+checklist%3F+In+Anna+Rogers%2C+Jordan+L%2E+Boyd%2DGraber%2C+and+Naoaki+Okazaki+%28eds%2E%29%2C+Findings+of+the+Associ)
34. Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, and Qing Wang. Clarifygpt: Empowering llm-based code generation with intention clarification, 2023. URL https://arxiv.org/abs/2310.10996. — [arXiv:2310.10996](https://arxiv.org/abs/2310.10996) [↗](https://arxiv.org/abs/2310.10996)
35. Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Yuanhong Zheng, Dongsheng Ma, Zirui Tang, Boyu Niu, Ziyang Miao, Hejun Dong, Siyi Qian, Jun… — [MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing](https://md.research.elamur.ai/free/en/papers/niu-2026-mineru2-5-decoupled-vision-language/)
36. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2024. URL https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774)
37. Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. Repograph: Enhancing ai software engineering with repository-level code graph, 2025. URL https://arxiv.org/abs/2410.14684. — [arXiv:2410.14684](https://arxiv.org/abs/2410.14684) [↗](https://arxiv.org/abs/2410.14684)
38. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Hugo Larochelle. Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program). J. Mach. Learn. Res., 22:164:1–164:20… [find](https://scholar.google.com/scholar?q=Joelle+Pineau%2C+Philippe+Vincent%2DLamarre%2C+Koustuv+Sinha%2C+Vincent+Larivi%C3%A8re%2C+Alina+Beygelzimer%2C+Florence+d%E2%80%99Alch%C3%A9%2DBuc%2C+Emily+B%2E+Fox%2C+and+Hugo+Larochelle%2E+Improving+reproducibility+in+machine+learning+res)
39. Karl Raimund Sir Popper. The logic of scientific discovery. Systematic Biology, 26:361, 1959. URL [https://philotextes.info/spip/IMG/pdf/](https://philotextes.info/spip/IMG/pdf/) popper-logic-scientific-discovery.pdf. [↗](https://philotextes.info/spip/IMG/pdf/)
40. Vignesh Prabhakar, Md Amirul Islam, Adam Atanas, Yao-Ting Wang, Joah Han, Aastha Jhunjhunwala, Rucha Apte, Robert Clark, Kang Xu, Zihan Wang, and Kai Liu. [Omniscience: A domain-specialized llm for scientific reasoning and discovery](https://md.research.elamur.ai/free/en/papers/prabhakar-2025-omniscience-domain-specialized-llm-scientific/), 2025. URL https://arxiv.org/abs/2503.17604. [↗](https://arxiv.org/abs/2503.17604)
41. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. [Large language models are zero shot hypothesis proposers](https://md.research.elamur.ai/free/en/papers/qi-2023-large-language-models-zero-shot/), 2023. URL [https://arxiv.org/](https://arxiv.org/) abs/2311.05965. [↗](https://arxiv.org/)
42. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. [Chatdev: Communicative agents for software development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62n…
43. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotti… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
44. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. [Agent laboratory: Using llm agents as research assistants](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agent-laboratory-using-llm/), 2025. URL https://arxiv.org/abs/2501.04227. [↗](https://arxiv.org/abs/2501.04227)
45. Minju Seo, Jinheon Baek, James Thorne, and Sung Ju Hwang. Retrieval-augmented data augmentation for low-resource domain tasks. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) rV1xeaaIKh. [↗](https://openreview.net/forum?id=)
46. Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Efficient long context language model retrieval with compression. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (… [find](https://scholar.google.com/scholar?q=Minju+Seo%2C+Jinheon+Baek%2C+Seongyun+Lee%2C+and+Sung+Ju+Hwang%2E+Efficient+long+context+language+model+retrieval+with+compression%2E+In+Wanxiang+Che%2C+Joyce+Nabende%2C+Ekaterina+Shutova%2C+and+Mohammad+Taher+Pilehv)
47. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. [Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/), 2024. URL https://arxiv.org/abs/ 2409.04109. [↗](https://arxiv.org/abs/)
48. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research, 2025. [URL https://arxiv.org/abs/2504.…](https://md.research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/) [↗](https://arxiv.org/abs/2504.…)
49. Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gers… — [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://md.research.elamur.ai/free/en/papers/tang-2023-ml-bench-evaluating-large-language/)
50. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625:476 – 482, 2024. URL [https://www.nature.com/articles/s41586-023-06747-5](https://www.nature.com/articles/s41586-023-06747-5). [↗](https://www.nature.com/articles/s41586-023-06747-5)
51. Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent llm framework for full-pipeline automl, 2024. URL https://arxiv.org/abs/2410.02958. — [arXiv:2410.02958](https://arxiv.org/abs/2410.02958) [↗](https://arxiv.org/abs/2410.02958)
52. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neur…
53. Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. [Mineru: An open-source solution for precise document content extraction](https://md.research.elamur.ai/free/en/papers/wang-2024-mineru-open-source-solution-precise/), 2024a. URL [https://arxiv.o…](https://arxiv.o…) [↗](https://arxiv.o…)
54. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. [Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aust…
55. OpenReview.net, 2024b. URL [https://openreview.net/forum?id=z8TW0ttBPp](https://openreview.net/forum?id=z8TW0ttBPp). [↗](https://openreview.net/forum?id=z8TW0ttBPp)
56. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Asso… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Anna+Roger)
57. Haoran Wei, Yaofeng Sun, and Yukun Li. [Deepseek-ocr: Contexts optical compression](https://md.research.elamur.ai/free/en/papers/wei-2025-deepseek-ocr-contexts-optical/). arXiv preprint arXiv:2510.18234, 2025.
58. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. [Cycleresearcher: Improving automated research via automated review](https://md.research.elamur.ai/free/en/papers/weng-2024-cycleresearcher-improving-automated-research/), 2025. URL https://arxiv.org/abs/2411.00816. [↗](https://arxiv.org/abs/2411.00816)
59. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. [Agentless: Demystifying llm-based software engineering agents](https://md.research.elamur.ai/free/en/papers/xia-2024-agentless-demystifying-llm-based/), 2024. URL https://arxiv.org/abs/2407.01489. [↗](https://arxiv.org/abs/2407.01489)
60. Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. [Scireplicate-bench: Benchmarking llms in agent-driven algorithmic reproduction from research papers](https://md.research.elamur.ai/free/en/papers/xiang-2025-scireplicate-bench-benchmarking-llms/), 2025. URL https://arxiv.org/abs/2504.00255. [↗](https://arxiv.org/abs/2504.00255)
61. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. [The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search](https://md.research.elamur.ai/free/en/papers/yamada-2025-ai-scientist-v2-workshop/), 2025. URL https://arxiv.org/abs/2504.08066. [↗](https://arxiv.org/abs/2504.08066)
62. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. [Large language models for automated open-domain scientific hypotheses discovery](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-automated/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Tha…
63. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
64. OpenReview.net, 2023. URL [https://openreview.net/forum?id=WE_vluYUL-X](https://openreview.net/forum?id=WE_vluYUL-X). [↗](https://openreview.net/forum?id=WE_vluYUL-X)
65. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. [Repocoder: Repository-level code completion through iterative retrieval and generation](https://md.research.elamur.ai/free/en/papers/zhang-2023-repocoder-repository-level-code-completion/). In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical…
66. Jiarui Zhang, Yuliang Liu, Zijun Wu, Guosheng Pang, Zhili Ye, Yupei Zhong, Junteng Ma, Tao Wei, Haiyang Xu, Weikai Chen, Zeen Wang, Qiangjun Ji, Fanxi Zhou, Qi Zhang, Yuanrui Hu, Jiahao Liu, Zhang Li, Ziyang Zhang, Qiang Liu, and Xiang Bai. Monkeyocr v1.5 technical report: Unlocking robust document… — [arXiv:2511.10390](https://arxiv.org/abs/2511.10390) [find](https://scholar.google.com/scholar?q=Jiarui+Zhang%2C+Yuliang+Liu%2C+Zijun+Wu%2C+Guosheng+Pang%2C+Zhili+Ye%2C+Yupei+Zhong%2C+Junteng+Ma%2C+Tao+Wei%2C+Haiyang+Xu%2C+Weikai+Chen%2C+Zeen+Wang%2C+Qiangjun+Ji%2C+Fanxi+Zhou%2C+Qi+Zhang%2C+Yuanrui+Hu%2C+Jiahao+Liu%2C+Zhang+Li%2C)
67. Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. [Mlcopilot: Unleashing the power of large language models in solving machine learning tasks](https://md.research.elamur.ai/free/en/papers/zhang-2024-mlcopilot-unleashing-power-large-language/). In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lingu…
68. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Morit…

*Updated 3 November 2026*