---
title: "Chain of Ideas — paper by Li et al. (2024) — summary and ideas"
description: "The authors propose the Chain-of-Ideas (CoI) agent, which imitates how people study the history of a… 33 references, cited by 47 papers here."
authors: ["Long Li", "Weiwen Xu", "Jiayan Guo", "Ruochen Zhao", "Xingxuan Li", "Yuqian Yuan", "Boqiang Zhang", "Yuming Jiang", "Yifei Xin", "Ronghao Dang", "Deli Zhao", "Yu Rong", "Tian Feng", "Lidong Bing"]
year: 2024
venue: "arXiv:2410.13185"
source: "https://arxiv.org/abs/2410.13185"
doi: "10.48550/arxiv.2410.13185"
arxiv: "2410.13185"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-chain-ideas-revolutionizing-research/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-chain-ideas-revolutionizing-research/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents

# Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents

**Authors:** [Long Li](https://md.research.elamur.ai/free/en/authors/long-li/), [Weiwen Xu](https://md.research.elamur.ai/free/en/authors/weiwen-xu/), [Jiayan Guo](https://md.research.elamur.ai/free/en/authors/jiayan-guo/), [Ruochen Zhao](https://md.research.elamur.ai/free/en/authors/ruochen-zhao/), [Xingxuan Li](https://md.research.elamur.ai/free/en/authors/xingxuan-li/), [Yuqian Yuan](https://md.research.elamur.ai/free/en/authors/yuqian-yuan/), Boqiang Zhang, Yuming Jiang, [Yifei Xin](https://md.research.elamur.ai/free/en/authors/yifei-xin/), Ronghao Dang, [Deli Zhao](https://md.research.elamur.ai/free/en/authors/deli-zhao/), [Yu Rong](https://md.research.elamur.ai/free/en/authors/yu-rong/), Tian Feng and [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) · **Year:** 2024 · **Venue:** arXiv:2410.13185 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.13185) · [PDF](https://arxiv.org/pdf/2410.13185) · [DOI 10.48550/arxiv.2410.13185](https://doi.org/10.48550/arxiv.2410.13185) · [arXiv 2410.13185](https://arxiv.org/abs/2410.13185) · [OpenAlex](https://openalex.org/W4403579374)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/li-2024-chain-ideas-revolutionizing-research/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Scientific literature grows so fast that researchers struggle to identify promising directions. Existing LLM ideation tools either prompt a model naively or fill its context with large, unorganized sets of papers. This can produce incoherent combinations of unrelated techniques. The authors propose the Chain-of-Ideas (CoI) agent, which imitates how people study the history of a field. It finds an anchor paper for a topic, then follows its references backward and its citing works forward. From these it builds a chain of ideas showing how the area evolved, and it builds several chains to cover different angles. The LLM reads each chain, predicts where the trend is heading, develops an idea step by step and checks its novelty against existing work. It then drafts and refines an experiment plan. For evaluation, the authors introduce Idea Arena, a pairwise tournament scored with ELO that agrees well with human judges. On AI-domain topics, CoI ranks first among automated systems and matches humans on novelty at low cost. The results suggest that organizing literature helps more than simply adding more of it.

## Key points
- The pipeline has three stages: CoI Construction (papers retrieved via Semantic Scholar), Idea Generation (trend prediction, idea consolidation and an iterative novelty-checker agent), and Experiment Design.
- Idea Arena compares ideas in pairs on Novelty, Significance, Clarity, Feasibility and Effectiveness. Experiment plans are compared on Feasibility, Quality and Clarity. The judges are GPT-4o, Claude-3.5-Sonnet, Gemini1.5-Pro-Exp0827 and human evaluators.
- The baselines are RAG, GPT-Researcher, ResearchAgent and AI-Scientist, with real papers as a human reference. In human evaluation, CoI beats the runner-up by 56 ELO points.
- Analysis shows that a clear developmental trend matters more for novel ideas than the number of related papers.
- The minimum cost is about $0.50 per idea plus experiment design. The code is available at github.com/DAMO-NLP-SG/CoI-Agent, with a demo on Hugging Face Spaces.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 207 works · 17,449 citations
- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 152 works · 114,627 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 236 works · 14,072 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 47 works · 6,996 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 37 works · 3,692 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works

## Cite

Cite as: Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng and Lidong Bing. 2024. Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents. arXiv:2410.13185.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2024chain,
  title = {{Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents}},
  author = {Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng and Lidong Bing},
  year = {2024},
  doi = {10.48550/arxiv.2410.13185},
  eprint = {2410.13185},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.13185}
}
```

- [← previous Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs?](https://md.research.elamur.ai/free/en/papers/li-2024-can-multiple-choice-questions-really/)
- [next → Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

33 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. Assafelovic. gpt-researcher, 2023. gpt-researcher. [find](https://scholar.google.com/scholar?q=Assafelovic%2E+gpt%2Dresearcher%2C+2023%2E+gpt%2Dresearcher%2E)
3. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. [Researchagent: Iterative research idea generation over scientific literature with large language models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/). arXiv preprint arXiv:2404.07738, 2024.
4. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Int… [find](https://scholar.google.com/scholar?q=Maciej+Besta%2C+Nils+Blach%2C+Ales+Kubicek%2C+Robert+Gerstenberger%2C+Michal+Podstawski%2C+Lukas+Gianinazzi%2C+Joanna+Gajda%2C+Tomasz+Lehmann%2C+Hubert+Niewiadomski%2C+Piotr+Nyczyk%2C+et+al%2E+Graph+of+thoughts%3A+Solving+el)
5. Daniil A Boiko, Robert MacKnight, and Gabe Gomes. [Emergent autonomous scientific research capabilities of large language models](https://md.research.elamur.ai/free/en/papers/boiko-2023-emergent-autonomous-scientific-research/). arXiv preprint arXiv:2304.05332, 2023.
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
7. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf). [↗](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)
8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
9. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/), 2024.
10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
11. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. [Complexity-based prompting for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2022-complexity-based-prompting-multi-step/). In The Eleventh International Conference on Learning Representations, 2022.
12. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. [MLAgentbench: Evaluating language agents on machine learning experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). In Forty-first International Conference on Machine Learning, 2024. URL [https://openreview.net/forum?id=1Fs1LvjYQW](https://openreview.net/forum?id=1Fs1LvjYQW). [↗](https://openreview.net/forum?id=1Fs1LvjYQW)
13. Hyunseung Kim, Jonggeol Na, and Won Bo Lee. Generative chemical transformer: neural machine learning of molecular geometric structures from chemical language via attention. Journal of chemical information and modeling, 61(12):5804–5814, 2021. [find](https://scholar.google.com/scholar?q=Hyunseung+Kim%2C+Jonggeol+Na%2C+and+Won+Bo+Lee%2E+Generative+chemical+transformer%3A+neural+machine+learning+of+molecular+geometric+structures+from+chemical+language+via+attention%2E+Journal+of+chemical+informa)
14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Advances in neural information processing systems, 35:22199–22213, 2022.
15. Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. [Can large language models unlock novel scientific research ideas?](https://md.research.elamur.ai/free/en/papers/kumar-2025-can-large-language-models-unlock/) arXiv preprint arXiv:2409.06185, 2024.
16. URL: — [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/) [https://github.com/assafelovic/Patrick](https://github.com/assafelovic/Patrick) Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proce… [↗](https://github.com/assafelovic/Patrick)
17. Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. [Mlr-copilot: Autonomous machine learning research based on large language models agents](https://md.research.elamur.ai/free/en/papers/li-2024-mlr-copilot-autonomous-machine/). arXiv preprint arXiv:2408.14033, 2024.
18. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292, 2024.
19. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. [Large language models are zero shot hypothesis proposers](https://md.research.elamur.ai/free/en/papers/qi-2023-large-language-models-zero-shot/). arXiv preprint arXiv:2311.05965, 2023.
20. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. [Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). arXiv preprint arXiv:2409.04109, 2024.
21. Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. [Graphgpt: Graph instruction tuning for large language models](https://md.research.elamur.ai/free/en/papers/tang-2023-graphgpt-graph-instruction-tuning-large/). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491–500, 2024.
22. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. [Scicode: A research coding benchmark curated by scientists](https://md.research.elamur.ai/free/en/papers/tian-2024-scicode-research-coding-benchmark/). arXiv preprint arXiv:2407.13168, 2024.
23. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. [Scimon: Scientific inspiration machines optimized for novelty](https://md.research.elamur.ai/free/en/papers/wang-2023-scimon-scientific-inspiration-machines/). arXiv preprint arXiv:2305.14259, 2023.
24. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171, 2022.
25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022.
26. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+et+al%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2C+2024a%2E)
27. Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. [Leandojo: Theorem proving with retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023…
28. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. [Large language models for automated open-domain scientific hypotheses discovery](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-automated/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 13545–135…
29. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Advances+in+Neural+Informati)
30. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
31. Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Deli Zhao, and Lidong Bing. Auto arena of llms: Automating llm evaluations with agent peer-battles and committee discussions. arXiv preprint arXiv:2405.20267, 2024. — [arXiv:2405.20267](https://arxiv.org/abs/2405.20267) [find](https://scholar.google.com/scholar?q=Ruochen+Zhao%2C+Wenxuan+Zhang%2C+Yew+Ken+Chia%2C+Deli+Zhao%2C+and+Lidong+Bing%2E+Auto+arena+of+llms%3A+Automating+llm+evaluations+with+agent+peer%2Dbattles+and+committee+discussions%2E+arXiv+preprint+arXiv%3A2405%2E20267)
32. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36, 2024.
33. Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020. [find](https://scholar.google.com/scholar?q=Jie+Zhou%2C+Ganqu+Cui%2C+Shengding+Hu%2C+Zhengyan+Zhang%2C+Cheng+Yang%2C+Zhiyuan+Liu%2C+Lifeng+Wang%2C+Changcheng+Li%2C+and+Maosong+Sun%2E+Graph+neural+networks%3A+A+review+of+methods+and+applications%2E+AI+open%2C+1%3A57%E2%80%9381%2C+)

*Updated 3 November 2026*