---
title: "Evaluating LLMs' Divergent Thinking Capabilities for Scientific"
description: "They release the generated ideas, scores, code and a continuously updated leaderboard. 83 references, cited by 13 papers here."
authors: ["Kai Ruan", "Xuan Wang", "Jixiang Hong", "Peng Wang", "Yang Liu", "Hao Sun"]
year: 2024
venue: "arXiv:2412.17596"
source: "https://arxiv.org/abs/2412.17596"
doi: "10.48550/arxiv.2412.17596"
arxiv: "2412.17596"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ruan-2024-evaluating-llms-divergent-thinking-capabilities/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ruan-2024-evaluating-llms-divergent-thinking-capabilities/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

# Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

**Authors:** Kai Ruan, [Xuan Wang](https://md.research.elamur.ai/free/en/authors/xuan-wang/), Jixiang Hong, [Peng Wang](https://md.research.elamur.ai/free/en/authors/peng-wang/), [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/) and [Hao Sun](https://md.research.elamur.ai/free/en/authors/hao-sun/) · **Year:** 2024 · **Venue:** arXiv:2412.17596 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.17596) · [PDF](https://arxiv.org/pdf/2412.17596) · [DOI 10.48550/arxiv.2412.17596](https://doi.org/10.48550/arxiv.2412.17596) · [arXiv 2412.17596](https://arxiv.org/abs/2412.17596) · [OpenAlex](https://openalex.org/W4405768265)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/ruan-2024-evaluating-llms-divergent-thinking-capabilities/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most benchmarks of LLMs on scientific tasks give models rich context, such as full papers or detailed task descriptions. They therefore reveal little about whether a model can produce research ideas from almost nothing. The authors address this gap with LiveIdeaBench. It prompts models with one scientific keyword and rates the resulting ideas on dimensions drawn from Guilford's divergent-thinking theory: originality, feasibility, clarity, fluency and flexibility. A rotating panel of strong LLM judges, selected from the LiveBench leaderboard, does the scoring. Safeguards prevent models from grading themselves and stop any single vendor from dominating the panel. The authors tested 41 models on 1,180 keywords from 22 fields. Scores on idea generation track general-intelligence rankings only loosely. For instance, the far smaller QwQ-32B-preview performs about as well as claude-3.7-sonnet:thinking. The authors conclude that scientific creativity in LLMs needs dedicated evaluation and perhaps different training strategies than general reasoning. They release the generated ideas, scores, code and a continuously updated leaderboard.

## Key points
- **Scale:** The benchmark covers 1,180 keywords across 22 scientific domains and evaluates 41 models from Anthropic, OpenAI, Google, Qwen, DeepSeek, Meta, Mistral and other developers, as ranked on LiveBench in March 2025.
- **Judging setup:**
  - The panel consists of the top 10 LiveBench models, with no more than 2 per organization.
  - Only one variant of each base model is kept, so o3-mini and o3-mini-high would not both serve.
  - The model being evaluated is always excluded from the panel.
  - Each idea receives originality, feasibility and clarity scores averaged over 3 randomly sampled judges, while fluency is assessed by a single sampled judge per keyword–model pair.
- **Flexibility metric:** It is computed by averaging the scores on the other four dimensions and taking the 30th percentile of those averages.
- **Generation protocol:**
  - Ideas target about 100 words, and any answer longer than 200 words is discarded.
  - Refused prompts get a second attempt with academic framing.
  - qwq-32b-preview required a special "Final Idea:" delimiter for parsing its output.
- **Resources:** The dataset is on Hugging Face as `6cf/liveideabench-v2`, the code is on GitHub at `x66ccff/liveideabench`, and both are also archived on Zenodo.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu and Hao Sun. 2024. Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context. arXiv:2412.17596.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ruan2024evaluating,
  title = {{Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context}},
  author = {Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu and Hao Sun},
  year = {2024},
  doi = {10.48550/arxiv.2412.17596},
  eprint = {2412.17596},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.17596}
}
```

- [← previous DevBench: A multimodal developmental benchmark for language learning](https://md.research.elamur.ai/free/en/papers/tan-2024-devbench-multimodal-developmental-benchmark-language/)
- [next → Evaluating Multilingual Long-Context Models for Retrieval and Reasoning](https://md.research.elamur.ai/free/en/papers/agrawal-2024-evaluating-multilingual-long-context-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

83 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wang, H. et al. Scientific discovery in the age of artificial intelligence. Nature 620, 47–60 (2023). [find](https://scholar.google.com/scholar?q=Wang%2C+H%2E+et+al%2E+Scientific+discovery+in+the+age+of+artificial+intelligence%2E+Nature+620%2C+47%E2%80%9360+%282023%29%2E)
2. Shi, F. & Evans, J. Surprising combinations of research contents and contexts are related to impact and emerge with scientific outsiders from distant disciplines. Nature Communications 14, 1641 (2023). [find](https://scholar.google.com/scholar?q=Shi%2C+F%2E+%26+Evans%2C+J%2E+Surprising+combinations+of+research+contents+and+contexts+are+related+to+impact+and+emerge+with+scientific+outsiders+from+distant+disciplines%2E+Nature+Communications+14%2C+1641+%282023%29)
3. AI4Science, M. R. & Quantum, M. A. The impact of large language models on scientific discovery: a preliminary study using GPT-4. arXiv preprint arXiv:2311.07361 (2023). — [arXiv:2311.07361](https://arxiv.org/abs/2311.07361) [find](https://scholar.google.com/scholar?q=AI4Science%2C+M%2E+R%2E+%26+Quantum%2C+M%2E+A%2E+The+impact+of+large+language+models+on+scientific+discovery%3A+a+preliminary+study+using+GPT%2D4%2E+arXiv+preprint+arXiv%3A2311%2E07361+%282023%29%2E+%E2%80%94+%5BarXiv%3A2311%2E07361%5D%28https%3A%2F%2Far)
4. Krenn, M. et al. Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network. Nature Machine Intelligence 5, 1326–1335 (2023). [find](https://scholar.google.com/scholar?q=Krenn%2C+M%2E+et+al%2E+Forecasting+the+future+of+artificial+intelligence+with+machine+learning%2Dbased+link+prediction+in+an+exponentially+growing+knowledge+network%2E+Nature+Machine+Intelligence+5%2C+1326%E2%80%931335+%28)
5. Rafner, J., Beaty, R. E., Kaufman, J. C., Lubart, T. & Sherson, J. Creativity in the age of generative AI. Nature Human Behaviour 7, 1836–1838 (2023). [find](https://scholar.google.com/scholar?q=Rafner%2C+J%2E%2C+Beaty%2C+R%2E+E%2E%2C+Kaufman%2C+J%2E+C%2E%2C+Lubart%2C+T%2E+%26+Sherson%2C+J%2E+Creativity+in+the+age+of+generative+AI%2E+Nature+Human+Behaviour+7%2C+1836%E2%80%931838+%282023%29%2E)
6. Trinh, T. H., Wu, Y., Le, Q. V., He, H. & Luong, T. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature 625, 476–482 (2024).
7. Getzels, J. W. & Jackson, P. W. Creativity and Intelligence: Exploration with Gifted Students (John Wiley & Sons, New York, 1962). [find](https://scholar.google.com/scholar?q=Getzels%2C+J%2E+W%2E+%26+Jackson%2C+P%2E+W%2E+Creativity+and+Intelligence%3A+Exploration+with+Gifted+Students+%28John+Wiley+%26+Sons%2C+New+York%2C+1962%29%2E)
8. Guilford, J. P. The nature of human intelligence (McGraw-Hill, 1967). [find](https://scholar.google.com/scholar?q=Guilford%2C+J%2E+P%2E+The+nature+of+human+intelligence+%28McGraw%2DHill%2C+1967%29%2E)
9. Jauk, E., Benedek, M., Dunst, B. & Neubauer, A. C. The relationship between intelligence and creativity: New support for the threshold hypothesis by means of empirical breakpoint detection. Intelligence 41, 212–221 (2013). URL [https://www.sciencedirect.com/science/](https://www.sciencedirect.com/science/) article/pii/S016028961300024X. [↗](https://www.sciencedirect.com/science/)
10. Kim, K. H. Can only intelligent people be creative? A meta-analysis. Journal of Secondary Gifted Education 16, 57–66 (2005). [find](https://scholar.google.com/scholar?q=Kim%2C+K%2E+H%2E+Can+only+intelligent+people+be+creative%3F+A+meta%2Danalysis%2E+Journal+of+Secondary+Gifted+Education+16%2C+57%E2%80%9366+%282005%29%2E)
11. Preckel, F., Holling, H. & Wiese, M. Relationship of intelligence and creativity in gifted and non-gifted students: An investigation of threshold theory. Personality and individual differences 40, 159–170 (2006). [find](https://scholar.google.com/scholar?q=Preckel%2C+F%2E%2C+Holling%2C+H%2E+%26+Wiese%2C+M%2E+Relationship+of+intelligence+and+creativity+in+gifted+and+non%2Dgifted+students%3A+An+investigation+of+threshold+theory%2E+Personality+and+individual+differences+40%2C+159)
12. Gralewski, J., Weremczuk, E. & Karwowski, M. Intelligence and creativity of polish middle-school students: Looking for the threshold hypothesis. The New Educational Review 29, 328–338 (2012). [find](https://scholar.google.com/scholar?q=Gralewski%2C+J%2E%2C+Weremczuk%2C+E%2E+%26+Karwowski%2C+M%2E+Intelligence+and+creativity+of+polish+middle%2Dschool+students%3A+Looking+for+the+threshold+hypothesis%2E+The+New+Educational+Review+29%2C+328%E2%80%93338+%282012%29%2E)
13. Park, G., Lubinski, D. & Benbow, C. P. Ability differences among people who have commensurate degrees matter for scientific creativity. Psychological science 19, 957–961 (2008). [find](https://scholar.google.com/scholar?q=Park%2C+G%2E%2C+Lubinski%2C+D%2E+%26+Benbow%2C+C%2E+P%2E+Ability+differences+among+people+who+have+commensurate+degrees+matter+for+scientific+creativity%2E+Psychological+science+19%2C+957%E2%80%93961+%282008%29%2E)
14. Runco, M. A. & Albert, R. S. The threshold theory regarding creativity and intelligence: An empirical test with gifted and nongifted children. Creative Child and Adult Quarterly 11, 212–218 (1986). [find](https://scholar.google.com/scholar?q=Runco%2C+M%2E+A%2E+%26+Albert%2C+R%2E+S%2E+The+threshold+theory+regarding+creativity+and+intelligence%3A+An+empirical+test+with+gifted+and+nongifted+children%2E+Creative+Child+and+Adult+Quarterly+11%2C+212%E2%80%93218+%281986%29%2E)
15. Benedek, M. On the relationship between creative potential and creative achievement: Challenges and future directions. Learning and Individual Differences 110, 102424 (2024). [find](https://scholar.google.com/scholar?q=Benedek%2C+M%2E+On+the+relationship+between+creative+potential+and+creative+achievement%3A+Challenges+and+future+directions%2E+Learning+and+Individual+Differences+110%2C+102424+%282024%29%2E)
16. Weiss, S., Steger, D., Schroeders, U. & Wilhelm, O. A reappraisal of the threshold hypothesis of creativity and intelligence. Journal of Intelligence 8, 38 (2020). [find](https://scholar.google.com/scholar?q=Weiss%2C+S%2E%2C+Steger%2C+D%2E%2C+Schroeders%2C+U%2E+%26+Wilhelm%2C+O%2E+A+reappraisal+of+the+threshold+hypothesis+of+creativity+and+intelligence%2E+Journal+of+Intelligence+8%2C+38+%282020%29%2E)
17. Guilford, J. P. Creativity. American Psychologist 5, 444–454 (1950). [find](https://scholar.google.com/scholar?q=Guilford%2C+J%2E+P%2E+Creativity%2E+American+Psychologist+5%2C+444%E2%80%93454+%281950%29%2E)
18. Boden, M. A. The creative mind: Myths and mechanisms (Routledge, 2004). [find](https://scholar.google.com/scholar?q=Boden%2C+M%2E+A%2E+The+creative+mind%3A+Myths+and+mechanisms+%28Routledge%2C+2004%29%2E)
19. Amabile, T. M. The social psychology of creativity: A componential conceptualization. Journal of Personality and Social Psychology, 45(2), 357 (1983). [find](https://scholar.google.com/scholar?q=Amabile%2C+T%2E+M%2E+The+social+psychology+of+creativity%3A+A+componential+conceptualization%2E+Journal+of+Personality+and+Social+Psychology%2C+45%282%29%2C+357+%281983%29%2E)
20. Rhodes, M. An analysis of creativity. The Phi Delta Kappan 42, 305–310 (1961). [find](https://scholar.google.com/scholar?q=Rhodes%2C+M%2E+An+analysis+of+creativity%2E+The+Phi+Delta+Kappan+42%2C+305%E2%80%93310+%281961%29%2E)
21. Cortes, R. A., Weinberger, A. B., Daker, R. J. & Green, A. E. Re-examining prominent measures of divergent and convergent creativity. Current Opinion in Behavioral Sciences 27, 90–93 (2019). URL [https://www.sciencedirect.com/science/article/pii/S2352154618301694](https://www.sciencedirect.com/science/article/pii/S2352154618301694). Creativity. [↗](https://www.sciencedirect.com/science/article/pii/S2352154618301694)
22. Rafner, J. et al. Towards game-based assessment of creative thinking. Creativity Research Journal, 35(4), 763–782 (2023). [find](https://scholar.google.com/scholar?q=Rafner%2C+J%2E+et+al%2E+Towards+game%2Dbased+assessment+of+creative+thinking%2E+Creativity+Research+Journal%2C+35%284%29%2C+763%E2%80%93782+%282023%29%2E)
23. Wei, J. et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems 35, 24824–24837 (2022).
24. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Proceedings of the 36th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2024).
25. Yao, S. et al. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2024). [find](https://scholar.google.com/scholar?q=Yao%2C+S%2E+et+al%2E+Tree+of+thoughts%3A+deliberate+problem+solving+with+large+language+models%2E+In+Proceedings+of+the+37th+International+Conference+on+Neural+Information+Processing+Systems+%28Red+Hook%2C+NY%2C+USA%2C)
26. Li, X.-Y., Xue, J.-T., Xie, Z. & Li, M. Think outside the code: Brainstorming boosts large language models in code generation. arXiv preprint arXiv:2305.10679 (2023). — [arXiv:2305.10679](https://arxiv.org/abs/2305.10679) [find](https://scholar.google.com/scholar?q=Li%2C+X%2E%2DY%2E%2C+Xue%2C+J%2E%2DT%2E%2C+Xie%2C+Z%2E+%26+Li%2C+M%2E+Think+outside+the+code%3A+Brainstorming+boosts+large+language+models+in+code+generation%2E+arXiv+preprint+arXiv%3A2305%2E10679+%282023%29%2E+%E2%80%94+%5BarXiv%3A2305%2E10679%5D%28https%3A%2F%2Farxi)
27. Lu, C. et al. [The AI Scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292 (2024).
28. Hu, X. et al. Nova: An iterative planning and search approach to enhance novelty and diversity of LLM generated ideas. arXiv preprint arXiv:2410.14255 (2024). [find](https://scholar.google.com/scholar?q=Hu%2C+X%2E+et+al%2E+Nova%3A+An+iterative+planning+and+search+approach+to+enhance+novelty+and+diversity+of+LLM+generated+ideas%2E+arXiv+preprint+arXiv%3A2410%2E14255+%282024%29%2E)
29. Baek, J., Jauhar, S. K., Cucerzan, S. & Hwang, S. J. [ResearchAgent: Iterative research idea generation over scientific literature with large language models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/). arXiv preprint arXiv:2404.07738 (2024).
30. Radensky, M. et al. Scideator: Human-LLM scientific idea generation grounded in research-paper facet recombination. arXiv preprint arXiv:2409.14634 (2024). — [arXiv:2409.14634](https://arxiv.org/abs/2409.14634) [find](https://scholar.google.com/scholar?q=Radensky%2C+M%2E+et+al%2E+Scideator%3A+Human%2DLLM+scientific+idea+generation+grounded+in+research%2Dpaper+facet+recombination%2E+arXiv+preprint+arXiv%3A2409%2E14634+%282024%29%2E+%E2%80%94+%5BarXiv%3A2409%2E14634%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
31. Gottweis, J. et al. Towards an AI co-scientist. arXiv preprint arXiv:2502.[18864 (2025)](https://md.research.elamur.ai/free/en/papers/gottweis-2025-accelerating-scientific-discovery-co/). URL https://arxiv.org/abs/2502.18864. 2502.18864. [↗](https://arxiv.org/abs/2502.18864)
32. Wang, Q., Downey, D., Ji, H. & Hope, T. Scimon: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 279–299 (2024). [find](https://scholar.google.com/scholar?q=Wang%2C+Q%2E%2C+Downey%2C+D%2E%2C+Ji%2C+H%2E+%26+Hope%2C+T%2E+Scimon%3A+Scientific+inspiration+machines+optimized+for+novelty%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
33. Gu, X. & Krenn, M. Interesting scientific idea generation using knowledge graphs and LLMs: Evaluations with 100 research group leaders. arXiv preprint arXiv:2405.17044 (2024). — [arXiv:2405.17044](https://arxiv.org/abs/2405.17044) [find](https://scholar.google.com/scholar?q=Gu%2C+X%2E+%26+Krenn%2C+M%2E+Interesting+scientific+idea+generation+using+knowledge+graphs+and+LLMs%3A+Evaluations+with+100+research+group+leaders%2E+arXiv+preprint+arXiv%3A2405%2E17044+%282024%29%2E+%E2%80%94+%5BarXiv%3A2405%2E17044%5D%28htt)
34. Gu, X. & Krenn, M. Forecasting high-impact research topics via machine learning on evolving knowledge graphs. arXiv preprint arXiv:2402.08640 (2024). — [arXiv:2402.08640](https://arxiv.org/abs/2402.08640) [find](https://scholar.google.com/scholar?q=Gu%2C+X%2E+%26+Krenn%2C+M%2E+Forecasting+high%2Dimpact+research+topics+via+machine+learning+on+evolving+knowledge+graphs%2E+arXiv+preprint+arXiv%3A2402%2E08640+%282024%29%2E+%E2%80%94+%5BarXiv%3A2402%2E08640%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E08)
35. Wang, W. et al. SciPIP: An LLM-based scientific paper idea proposer. arXiv preprint arXiv:2410.23166 (2024). — [arXiv:2410.23166](https://arxiv.org/abs/2410.23166) [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E+et+al%2E+SciPIP%3A+An+LLM%2Dbased+scientific+paper+idea+proposer%2E+arXiv+preprint+arXiv%3A2410%2E23166+%282024%29%2E+%E2%80%94+%5BarXiv%3A2410%2E23166%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E23166%29)
36. Pu, K. et al. IdeaSynth: Iterative research idea development through evolving and composing idea facets with literature-grounded feedback. arXiv preprint arXiv:2410.04025 (2024). — [arXiv:2410.04025](https://arxiv.org/abs/2410.04025) [find](https://scholar.google.com/scholar?q=Pu%2C+K%2E+et+al%2E+IdeaSynth%3A+Iterative+research+idea+development+through+evolving+and+composing+idea+facets+with+literature%2Dgrounded+feedback%2E+arXiv+preprint+arXiv%3A2410%2E04025+%282024%29%2E+%E2%80%94+%5BarXiv%3A2410%2E04025%5D%28)
37. Meincke, L., Girotra, K., Nave, G., Terwiesch, C. & Ulrich, K. T. Using large language models for idea generation in innovation. The Wharton School Research Paper Forthcoming (2024). URL [https://ssrn.com/abstract=4526071](https://ssrn.com/abstract=4526071). 4526071. [↗](https://ssrn.com/abstract=4526071)
38. Si, C., Yang, D. & Hashimoto, T. [Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). arXiv preprint arXiv:2409.04109 (2024). URL https://arxiv.org/abs/2409.04109. 2409.04109. [↗](https://arxiv.org/abs/2409.04109)
39. Lee, B. C. & Chung, J. An empirical investigation of the impact of ChatGPT on creativity. Nature Human Behaviour 8, 1906–1914 (2024). [find](https://scholar.google.com/scholar?q=Lee%2C+B%2E+C%2E+%26+Chung%2C+J%2E+An+empirical+investigation+of+the+impact+of+ChatGPT+on+creativity%2E+Nature+Human+Behaviour+8%2C+1906%E2%80%931914+%282024%29%2E)
40. Cropley, D. Is artificial intelligence more creative than humans?: ChatGPT and the divergent association task. Learning Letters 2, 13–13 (2023). [find](https://scholar.google.com/scholar?q=Cropley%2C+D%2E+Is+artificial+intelligence+more+creative+than+humans%3F%3A+ChatGPT+and+the+divergent+association+task%2E+Learning+Letters+2%2C+13%E2%80%9313+%282023%29%2E)
41. Marrone, R., Cropley, D. & Medeiros, K. How does narrow AI impact human creativity? Creativity Research Journal 1–11 (2024). [find](https://scholar.google.com/scholar?q=Marrone%2C+R%2E%2C+Cropley%2C+D%2E+%26+Medeiros%2C+K%2E+How+does+narrow+AI+impact+human+creativity%3F+Creativity+Research+Journal+1%E2%80%9311+%282024%29%2E)
42. Wenger, E. & Kenett, Y. We’re different, we’re the same: Creative homogeneity across LLMs. arXiv preprint arXiv:2501.19361 (2025). — [arXiv:2501.19361](https://arxiv.org/abs/2501.19361) [find](https://scholar.google.com/scholar?q=Wenger%2C+E%2E+%26+Kenett%2C+Y%2E+We%E2%80%99re+different%2C+we%E2%80%99re+the+same%3A+Creative+homogeneity+across+LLMs%2E+arXiv+preprint+arXiv%3A2501%2E19361+%282025%29%2E+%E2%80%94+%5BarXiv%3A2501%2E19361%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E19361%29)
43. Doshi, A. R. & Hauser, O. P. Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances 10 (2023). URL [https://api.semanticscholar.org/CorpusID:271119565](https://api.semanticscholar.org/CorpusID:271119565). [↗](https://api.semanticscholar.org/CorpusID:271119565)
44. Lu, Y., Wang, D., Li, T., Jiang, D. & Khashabi, D. [Benchmarking language model creativity: A case study on code generation](https://md.research.elamur.ai/free/en/papers/lu-2025-benchmarking-language-model-creativity-case/). arXiv preprint arXiv:2407.09007 (2024).
45. Zhao, Y. et al. Assessing and understanding creativity in large language models. arXiv preprint arXiv:2401.12491 (2024). — [arXiv:2401.12491](https://arxiv.org/abs/2401.12491) [find](https://scholar.google.com/scholar?q=Zhao%2C+Y%2E+et+al%2E+Assessing+and+understanding+creativity+in+large+language+models%2E+arXiv+preprint+arXiv%3A2401%2E12491+%282024%29%2E+%E2%80%94+%5BarXiv%3A2401%2E12491%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2401%2E12491%29)
46. Dubois, Y., Galambosi, B., Liang, P. & Hashimoto, T. B. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 (2024). [find](https://scholar.google.com/scholar?q=Dubois%2C+Y%2E%2C+Galambosi%2C+B%2E%2C+Liang%2C+P%2E+%26+Hashimoto%2C+T%2E+B%2E+Length%2Dcontrolled+AlpacaEval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475+%282024%29%2E)
47. Zheng, L. et al. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems 36, 46595–46623 (2023).
48. Li, T. et al. [From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). arXiv preprint arXiv:2406.11939 (2024).
49. Chiang, C.-H. & Lee, H.-y. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 15607–15631 (Association for Computational Linguistics, Toronto, Canada, 2023). URL https:/… [find](https://scholar.google.com/scholar?q=Chiang%2C+C%2E%2DH%2E+%26+Lee%2C+H%2E%2Dy%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+)
50. Chen, Y., Wang, R., Jiang, H., Shi, S. & Xu, R. [Exploring the use of large language models for reference-free text quality evaluation: An empirical study](https://md.research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/). In Park, J. C. et al. (eds.) Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), 361–374 (Association for Com…
51. Huang, F., Kwak, H., Park, K. & An, J. ChatGPT rates natural language explanation quality like humans: But on which scales? In Calzolari, N. et al. (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 3111–3… [find](https://scholar.google.com/scholar?q=Huang%2C+F%2E%2C+Kwak%2C+H%2E%2C+Park%2C+K%2E+%26+An%2C+J%2E+ChatGPT+rates+natural+language+explanation+quality+like+humans%3A+But+on+which+scales%3F+In+Calzolari%2C+N%2E+et+al%2E+%28eds%2E%29+Proceedings+of+the+2024+Joint+International+C)
52. Gilardi, F., Alizadeh, M. & Kubli, M. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120, e2305016120 (2023). — [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
53. Badshah, S. & Sajjad, H. Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text. arXiv preprint arXiv:2408.09235 (2024). — [arXiv:2408.09235](https://arxiv.org/abs/2408.09235) [find](https://scholar.google.com/scholar?q=Badshah%2C+S%2E+%26+Sajjad%2C+H%2E+Reference%2DGuided+Verdict%3A+LLMs%2Das%2DJudges+in+Automatic+Evaluation+of+Free%2DForm+Text%2E+arXiv+preprint+arXiv%3A2408%2E09235+%282024%29%2E+%E2%80%94+%5BarXiv%3A2408%2E09235%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E092)
54. Verga, P. et al. [Replacing judges with juries: Evaluating LLM generations with a panel of diverse models](https://md.research.elamur.ai/free/en/papers/verga-2024-replacing-judges-juries-evaluating-llm/). arXiv preprint arXiv:2404.18796 (2024).
55. Kim, S. et al. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv:2405.01535 (2024).
56. Rafner, J. et al. Mapping citizen science through the lens of human-centered AI. Human Computation 9, 66–95 (2022). [find](https://scholar.google.com/scholar?q=Rafner%2C+J%2E+et+al%2E+Mapping+citizen+science+through+the+lens+of+human%2Dcentered+AI%2E+Human+Computation+9%2C+66%E2%80%9395+%282022%29%2E)
57. Akata, Z. et al. A research agenda for hybrid intelligence: augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer 53, 18–28 (2020). [find](https://scholar.google.com/scholar?q=Akata%2C+Z%2E+et+al%2E+A+research+agenda+for+hybrid+intelligence%3A+augmenting+human+intellect+with+collaborative%2C+adaptive%2C+responsible%2C+and+explainable+artificial+intelligence%2E+Computer+53%2C+18%E2%80%9328+%282020%29%2E)
58. Dellermann, D., Ebel, P., Söllner, M. & Leimeister, J. M. Hybrid intelligence. Business & Information Systems Engineering 61, 637–643 (2019). [find](https://scholar.google.com/scholar?q=Dellermann%2C+D%2E%2C+Ebel%2C+P%2E%2C+S%C3%B6llner%2C+M%2E+%26+Leimeister%2C+J%2E+M%2E+Hybrid+intelligence%2E+Business+%26+Information+Systems+Engineering+61%2C+637%E2%80%93643+%282019%29%2E)
59. White, C. et al. LiveBench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations (2025). URL [https://openreview.net/forum?id=sKYHBTAxVa](https://openreview.net/forum?id=sKYHBTAxVa). [↗](https://openreview.net/forum?id=sKYHBTAxVa)
60. Jain, N. et al. [LiveCodeBench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974 (2024).
61. Wei, J., Huang, D., Lu, Y., Zhou, D. & Le, Q. V. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958 (2024). URL [https://arxiv.org/](https://arxiv.org/) abs/2308.03958. 2308.03958. [↗](https://arxiv.org/)
62. Gu, J. et al. A survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594 (2025). URL https://arxiv.org/abs/2411.15594. 2411.15594. [↗](https://arxiv.org/abs/2411.15594)
63. Fanous, A. et al. SycEval: Evaluating LLM sycophancy. arXiv preprint arXiv:2502.08177 (2025). URL https://arxiv.org/abs/2502.08177. 2502.08177. — [arXiv:2502.08177](https://arxiv.org/abs/2502.08177) [↗](https://arxiv.org/abs/2502.08177)
64. Huang, L. et al. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Transactions on Information Systems 43, 1–55 (2025).
65. Falk, J. et al. How do hackathons foster creativity? towards ai collaborative evaluation of creativity at scale. In CHI’25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery (ACM), 2025). [find](https://scholar.google.com/scholar?q=Falk%2C+J%2E+et+al%2E+How+do+hackathons+foster+creativity%3F+towards+ai+collaborative+evaluation+of+creativity+at+scale%2E+In+CHI%E2%80%9925%3A+Proceedings+of+the+2025+CHI+Conference+on+Human+Factors+in+Computing+Systems)
66. DiStefano, P. V. et al. Evaluating ai’s ideas: The role of individual creativity and expertise in human-ai co-creativity (2025). [find](https://scholar.google.com/scholar?q=DiStefano%2C+P%2E+V%2E+et+al%2E+Evaluating+ai%E2%80%99s+ideas%3A+The+role+of+individual+creativity+and+expertise+in+human%2Dai+co%2Dcreativity+%282025%29%2E)
67. Sourati, J. & Evans, J. A. Accelerating science with human-aware artificial intelligence. Nature Human Behaviour 7, 1682–1696 (2023). [find](https://scholar.google.com/scholar?q=Sourati%2C+J%2E+%26+Evans%2C+J%2E+A%2E+Accelerating+science+with+human%2Daware+artificial+intelligence%2E+Nature+Human+Behaviour+7%2C+1682%E2%80%931696+%282023%29%2E)
68. Lacoste, A., Luccioni, A., Schmidt, V. & Dandres, T. [Quantifying the carbon emissions of machine learning](https://md.research.elamur.ai/free/en/papers/lacoste-2019-quantifying-carbon-emissions-machine-learning/). arXiv preprint arXiv:1910.09700 (2019).
69. Rince, S. & Banse, A. EcoLogits: Evaluate the environmental impacts of generative AI (2025). URL https://doi.org/10.5281/zenodo.15132099. — [doi:10.5281/zenodo.15132099](https://doi.org/10.5281/zenodo.15132099) [↗](https://doi.org/10.5281/zenodo.15132099)
70. AI Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Claude-3 Model Card (2024). [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+The+Claude+3+model+family%3A+Opus%2C+Sonnet%2C+Haiku%2E+Claude%2D3+Model+Card+%282024%29%2E)
71. Achiam, J. et al. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774 (2023).
72. Team, G. et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530 (2024).
73. Bai, J. et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609 (2023).
74. Qwen: et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2025). URL https://arxiv.org/abs/2412.15115. 2412.15115. [↗](https://arxiv.org/abs/2412.15115)
75. Liu, A. et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024). [find](https://scholar.google.com/scholar?q=Liu%2C+A%2E+et+al%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2E+arXiv+preprint+arXiv%3A2405%2E04434+%282024%29%2E)
76. DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437 (2025). 2412. 19437. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+DeepSeek%2DV3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437+%282025%29%2E+2412%2E+19437%2E)
77. DeepSeek-AI. [DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948 (2025). 2501.12948.
78. Dubey, A. et al. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783 (2024).
79. Jiang, A. Q. et al. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023). [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E+et+al%2E+Mistral+7B%2E+arXiv+preprint+arXiv%3A2310%2E06825+%282023%29%2E)
80. Intelligence, A. A. G. The Amazon Nova family of models: Technical report and model card (2024). [find](https://scholar.google.com/scholar?q=Intelligence%2C+A%2E+A%2E+G%2E+The+Amazon+Nova+family+of+models%3A+Technical+report+and+model+card+%282024%29%2E)
81. Abdin, M. et al. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905 (2024). 2412.08905.
82. Beltagy, I., Lo, K. & Cohan, A. [SciBERT: A pretrained language model for scientific text](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/). In Inui, K., Jiang, J., Ng, V. & Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (…
83. Cohen, E. The boundary lens: Theorising academic activity. In The University and its Boundaries: Thriving or Surviving in the 21st Century, 14–41 (Routledge, New York, New York, 2021), 1 edn. Acknowledgements: The work is supported by the National Natural Science Foundation of China (No. 92270118, N… [find](https://scholar.google.com/scholar?q=Cohen%2C+E%2E+The+boundary+lens%3A+Theorising+academic+activity%2E+In+The+University+and+its+Boundaries%3A+Thriving+or+Surviving+in+the+21st+Century%2C+14%E2%80%9341+%28Routledge%2C+New+York%2C+New+York%2C+2021%29%2C+1+edn%2E+Acknowle)

*Updated 31 October 2026*