---
title: "ResearchBench — paper by Liu et al. (2025)"
description: "Scientific hypothesis generation has had no dedicated benchmark for LLMs. 32 references, cited by 23 papers here."
authors: ["Yujie Liu", "Zonglin Yang", "Tong Xie", "Jinjie Ni", "Ben Gao", "Yuqiang Li", "Shixiang Tang", "Wanli Ouyang", "Erik Cambria", "Dongzhan Zhou"]
year: 2025
venue: "arXiv:2503.21248"
source: "https://arxiv.org/abs/2503.21248"
doi: "10.48550/arxiv.2503.21248"
arxiv: "2503.21248"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2025-researchbench-benchmarking-llms-scientific/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2025-researchbench-benchmarking-llms-scientific/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

# ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

**Authors:** [Yujie Liu](https://md.research.elamur.ai/free/en/authors/yujie-liu/), [Zonglin Yang](https://md.research.elamur.ai/free/en/authors/zonglin-yang/), [Tong Xie](https://md.research.elamur.ai/free/en/authors/tong-xie/), [Jinjie Ni](https://md.research.elamur.ai/free/en/authors/jinjie-ni/), [Ben Gao](https://md.research.elamur.ai/free/en/authors/ben-gao/), [Yuqiang Li](https://md.research.elamur.ai/free/en/authors/yuqiang-li/), [Shixiang Tang](https://md.research.elamur.ai/free/en/authors/shixiang-tang/), [Wanli Ouyang](https://md.research.elamur.ai/free/en/authors/wanli-ouyang/), [Erik Cambria](https://md.research.elamur.ai/free/en/authors/erik-cambria/) and [Dongzhan Zhou](https://md.research.elamur.ai/free/en/authors/dongzhan-zhou/) · **Year:** 2025 · **Venue:** arXiv:2503.21248 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.21248) · [PDF](https://arxiv.org/pdf/2503.21248) · [DOI 10.48550/arxiv.2503.21248](https://doi.org/10.48550/arxiv.2503.21248) · [arXiv 2503.21248](https://arxiv.org/abs/2503.21248) · [OpenAlex](https://openalex.org/W4415062263)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/liu-2025-researchbench-benchmarking-llms-scientific/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Scientific hypothesis generation has had no dedicated benchmark for LLMs. ResearchBench addresses this by building on a decomposition from MOOSE-Chem. In that view, a hypothesis comes from combining a research background with one or more inspirations. Discovery can then be split into three sub-tasks: finding inspirations, composing a hypothesis from them, and ranking candidate hypotheses. Solving all three perfectly would amount to solving discovery itself.

The authors collected 1,386 papers from 12 disciplines, all published in 2024 or later in top venues, to limit pretraining contamination. An LLM-based agentic pipeline breaks each paper into its question, background survey, inspirations and main hypothesis. It uses self-refinement and checks that the extracted inspirations are necessary and sufficient. Because the pipeline is automated, the benchmark can be refreshed as model cutoffs advance.

Experiments show LLMs are surprisingly good at the out-of-distribution task of retrieving inspirations, even with hard negatives. They do only moderately at composition and ranking, and ranking suffers from strong order bias. The authors argue that LLMs could serve as "research hypothesis mines."

## Key points
- The corpus has 1,386 papers across 12 disciplines, including Chemistry, Physics, Law, Math and Astronomy, all from 2024 onward.
- Five experts reviewed 62 sampled papers. Extraction accuracy was 91.9% counting only major issues and 82.3% when minor issues were included.
- When GPT-4o picks the top 4% of inspiration candidates, a ground-truth inspiration is among them 45.7% of the time.
- Pairwise ranking is distorted by input order. Llama-3.1-8B gives self-contradictory judgments 91.67% of the time, compared with 19.17% for Claude 3.5 Sonnet.
- In the error analysis, 56% of retrieval failures came from matching surface title or abstract overlap, and 52% of composition failures came from omitting key elements.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 221 works · 39,953 citations
- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 213 works · 22,907 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 193 works · 27,739 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 164 works · 13,544 citations
- [Wuhan University](https://md.research.elamur.ai/free/en/universities/wuhan-university/) — University, Wuhan, China · 52 works · 1,456 citations
- [UNSW Sydney](https://md.research.elamur.ai/free/en/universities/unsw-sydney/) — University, Sydney, Australia · 40 works · 1,630 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works

## Cite

Cite as: Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria and Dongzhan Zhou. 2025. ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition. arXiv:2503.21248.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2025researchbench,
  title = {{ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition}},
  author = {Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria and Dongzhan Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2503.21248},
  eprint = {2503.21248},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.21248}
}
```

- [← previous Removal of Hallucination on Hallucination: Debate-Augmented RAG](https://md.research.elamur.ai/free/en/papers/hu-2025-removal-hallucination-hallucination-debate-augmented/)
- [next → Seed-Coder: Let the Code Model Curate Data for Itself](https://md.research.elamur.ai/free/en/papers/bytedance-seed-2025-seed-coder-let-code/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

32 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mathias Benedek, Tanja Könen, and Aljoscha C Neubauer. 2012. Associative abilities underlying creativity. Psychology of Aesthetics, Creativity, and the Arts, 6(3):273. [find](https://scholar.google.com/scholar?q=Mathias+Benedek%2C+Tanja+K%C3%B6nen%2C+and+Aljoscha+C+Neubauer%2E+2012%2E+Associative+abilities+underlying+creativity%2E+Psychology+of+Aesthetics%2C+Creativity%2C+and+the+Arts%2C+6%283%29%3A273%2E)
2. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2024. [Scienceagentbench: Toward rigorous assessme…](https://md.research.elamur.ai/free/en/papers/chen-2024-scienceagentbench-rigorous-assessment-language/)
3. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). In Forty-first International Conference on…
4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arXiv prep…
5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
6. Aniketh Garikaparthi, Manasi Patwardhan, Aditya Sanjiv Kanade, Aman Hassan, Lovekesh Vig, and Arman Cohan. 2025. Mir: Methodology inspiration retrieval for scientific research problems. arXiv preprint arXiv:2506.00249. [find](https://scholar.google.com/scholar?q=Aniketh+Garikaparthi%2C+Manasi+Patwardhan%2C+Aditya+Sanjiv+Kanade%2C+Aman+Hassan%2C+Lovekesh+Vig%2C+and+Arman+Cohan%2E+2025%2E+Mir%3A+Methodology+inspiration+retrieval+for+scientific+research+problems%2E+arXiv+preprint)
7. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. [Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures](https://md.research.elamur.ai/free/en/papers/ni-2024-mixeval-deriving-wisdom-crowd-llm/). arXiv preprint arXiv:2406.06565.
8. Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, and Aidong Zhang. 2024. Ideabench: Benchmarking large language models for research idea generation. CoRR, abs/2411.02429. — [arXiv:2411.02429](https://arxiv.org/abs/2411.02429) [find](https://scholar.google.com/scholar?q=Sikun+Guo%2C+Amir+Hassan+Shariatmadari%2C+Guangzhi+Xiong%2C+Albert+Huang%2C+Eric+Xie%2C+Stefan+Bekiranov%2C+and+Aidong+Zhang%2E+2024%2E+Ideabench%3A+Benchmarking+large+language+models+for+research+idea+generation%2E+CoRR)
9. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. [Alphaevolve: A coding agent for scientific and algorithmic discovery](https://md.research.elamur.ai/free/en/papers/novikov-2025-alphaevolve-coding-agent-scientific-algorithmic/). arXiv preprint arXiv:2506.13131.
10. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
11. Arthur Koestler. 1964. The act of creation. London: Hutchinson. [find](https://scholar.google.com/scholar?q=Arthur+Koestler%2E+1964%2E+The+act+of+creation%2E+London%3A+Hutchinson%2E)
12. Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. 2024. [Can large language models unlock novel scientific research ideas?](https://md.research.elamur.ai/free/en/papers/kumar-2025-can-large-language-models-unlock/) CoRR, abs/2409.06185.
13. Byung Cheol Lee and Jaeyeon Chung. 2024. An empirical investigation of the impact of chatgpt on creativity. Nature Human Behaviour, pages 1–9. [find](https://scholar.google.com/scholar?q=Byung+Cheol+Lee+and+Jaeyeon+Chung%2E+2024%2E+An+empirical+investigation+of+the+impact+of+chatgpt+on+creativity%2E+Nature+Human+Behaviour%2C+pages+1%E2%80%939%2E)
14. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2E)
15. Wanhao Liu, Zonglin Yang, Jue Wang, Lidong Bing, Di Zhang, Dongzhan Zhou, Yuqiang Li, Houqiang Li, Erik Cambria, and Wanli Ouyang. 2025. Moose-chem3: Toward experiment-guided hypothesis ranking via simulated experimental feedback. arXiv preprint arXiv:2505.17873. [find](https://scholar.google.com/scholar?q=Wanhao+Liu%2C+Zonglin+Yang%2C+Jue+Wang%2C+Lidong+Bing%2C+Di+Zhang%2C+Dongzhan+Zhou%2C+Yuqiang+Li%2C+Houqiang+Li%2C+Erik+Cambria%2C+and+Wanli+Ouyang%2E+2025%2E+Moose%2Dchem3%3A+Toward+experiment%2Dguided+hypothesis+ranking+via+si)
16. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. 2025. [Llm4sr: A survey on large language models for scientific research](https://md.research.elamur.ai/free/en/papers/luo-2025-llm4sr-survey-large-language/). arXiv preprint arXiv:2501.04306.
17. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
18. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2024. [Discoverybench: Towards data-driven discovery with large language models](https://md.research.elamur.ai/free/en/papers/majumder-2024-discoverybench-data-driven-discovery/). CoRR, abs/2407.01725.
19. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). Nature, 625(7995):468–475.
20. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. 2025a. [Llm-sr: Scientific equation discovery via programming with large language models](https://md.research.elamur.ai/free/en/papers/shojaee-2024-llm-sr-scientific-equation/). In The Thirteenth International Conference on Learning Representations.
21. Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. 2025b. [Llm-srbench: A new benchmark for scientific equation discovery with large language models](https://md.research.elamur.ai/free/en/papers/shojaee-2025-llm-srbench-new-benchmark/). arXiv preprint arXiv:2504.10415.
22. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. [Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). arXiv preprint arXiv:2409.04109.
23. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. 2025. [Paperbench: Evaluating ai’s ability to replicate ai research](https://md.research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/). arXiv preprint arXiv:2504.01848.
24. Anthropic Team. 2024. The claude 3 model family: Opus, sonnet, haiku. [find](https://scholar.google.com/scholar?q=Anthropic+Team%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E)
25. Qwen Team et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2:3. [find](https://scholar.google.com/scholar?q=Qwen+Team+et+al%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2C+2%3A3%2E)
26. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. [Scimon: Scientific inspiration machines optimized for novelty](https://md.research.elamur.ai/free/en/papers/wang-2023-scimon-scientific-inspiration-machines/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 279– 299.…
27. Yixuan Weng, Minjun Zhu, Qiujie Xie, Qiyao Sun, Zhen Lin, Sifan Liu, and Yue Zhang. 2025. [Deepscientist: Advancing frontier-pushing scientific findings progressively](https://md.research.elamur.ai/free/en/papers/weng-2025-deepscientist-advancing-frontier-pushing/). arXiv preprint arXiv:2509.26603.
28. Zonglin Yang and Lidong Bing. 2026. Moose-star: Unlocking tractable training for scientific discovery by breaking the complexity barrier. arXiv preprint arXiv:2603.03756. [find](https://scholar.google.com/scholar?q=Zonglin+Yang+and+Lidong+Bing%2E+2026%2E+Moose%2Dstar%3A+Unlocking+tractable+training+for+scientific+discovery+by+breaking+the+complexity+barrier%2E+arXiv+preprint+arXiv%3A2603%2E03756%2E)
29. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. [Large language models for automated open-domain scientific hypotheses discovery](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-automated/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 13545–13565, Bangkok, Thailand. Association for Comput…
30. Zonglin Yang, Wanhao Liu, Ben Gao, Yujie Liu, Wei Li, Tong Xie, Lidong Bing, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. 2025a. Moose-chem2: Exploring llm limits in fine-grained scientific hypothesis discovery via hierarchical search. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zonglin+Yang%2C+Wanhao+Liu%2C+Ben+Gao%2C+Yujie+Liu%2C+Wei+Li%2C+Tong+Xie%2C+Lidong+Bing%2C+Wanli+Ouyang%2C+Erik+Cambria%2C+and+Dongzhan+Zhou%2E+2025a%2E+Moose%2Dchem2%3A+Exploring+llm+limits+in+fine%2Dgrained+scientific+hypothes)
31. Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. 2025b. [Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses](https://md.research.elamur.ai/free/en/papers/yang-2024-moose-chem-large-language/). In Proceedings of the International Conference on Learning Representations (ICLR)…
32. James Webb Young. 1975. A technique for producing ideas. [find](https://scholar.google.com/scholar?q=James+Webb+Young%2E+1975%2E+A+technique+for+producing+ideas%2E)

*Updated 3 November 2026*