---
title: "SCI-IDEA — paper by Keya et al. (2025) — summary and ideas"
description: "SCI-IDEA tackles a hard problem for LLM-based research assistants: proposing scientific ideas that are… 7 references, cited by 4 papers here."
authors: ["Farhana Keya", "Gollam Rabby", "Prasenjit Mitra", "Sahar Vahdati", "Soren Auer", "Yaser Jaradeh"]
year: 2025
venue: "arXiv:2503.19257"
source: "https://arxiv.org/abs/2503.19257"
doi: "10.48550/arxiv.2503.19257"
arxiv: "2503.19257"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/keya-2025-sci-idea-context-aware-scientific/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/keya-2025-sci-idea-context-aware-scientific/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings

# SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings

**Authors:** Farhana Keya, [Gollam Rabby](https://md.research.elamur.ai/free/en/authors/gollam-rabby/), [Prasenjit Mitra](https://md.research.elamur.ai/free/en/authors/prasenjit-mitra/), Sahar Vahdati, [Soren Auer](https://md.research.elamur.ai/free/en/authors/soren-auer/) and Yaser Jaradeh · **Year:** 2025 · **Venue:** arXiv:2503.19257 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.19257) · [PDF](https://arxiv.org/pdf/2503.19257) · [DOI 10.48550/arxiv.2503.19257](https://doi.org/10.48550/arxiv.2503.19257) · [arXiv 2503.19257](https://arxiv.org/abs/2503.19257) · [OpenAlex](https://openalex.org/W6966984688)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/keya-2025-sci-idea-context-aware-scientific/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
SCI-IDEA tackles a hard problem for LLM-based research assistants: proposing scientific ideas that are novel, grounded in context, and actually workable. Starting from a researcher's identifier (such as an ORCID ID) or a free-text query, the framework pulls related publications from CORE, arXiv and Semantic Scholar. It then uses LLMs to extract structured facets (objectives, methodologies, evaluation, future work) and infers research gaps from them. Candidate ideas come from zero-shot, zero-shot chain-of-thought or few-shot prompting. Each idea gets two scores: novelty, based on embedding cosine dissimilarity, and surprise, based on negative log-likelihood under a pretrained language model. Ideas that clear both thresholds are flagged as "Aha moments" and refined further with researcher feedback. The authors compare token-level and sentence-level embeddings across GPT-4o, GPT-4.5, DeepSeek-32B and DeepSeek-70B. LLM judges and human experts rate ideas on novelty, excitement, feasibility and effectiveness. The best configurations average roughly 6.8–6.9 out of 10. The work matters as a structured, human-in-the-loop pipeline for ideation, although human ratings show that LLM judges can be overly generous.

## Key points
- **Pipeline:** facet extraction, research-gap detection, prompted idea generation, then embedding-based "Aha Moment" filtering. The paper's example uses a novelty threshold θn = 0.7 and a surprise threshold θs = 2.0.
- **Token-level embeddings:** the best per-model averages are 6.84 (GPT-4o, 2-shot), 6.86 (GPT-4.5, 2-shot), 6.89 (DeepSeek-32B, 2-shot) and 6.84 (DeepSeek-70B, 3-shot).
- **Sentence-level embeddings:** the best per-model averages are 6.87 (GPT-4o, 5-shot), 6.86 (GPT-4.5, 3-shot), 6.83 (DeepSeek-32B, zero-shot CoT) and 6.87 (DeepSeek-70B, 5-shot).
- **Prompting and model size:** few-shot prompting usually beats zero-shot variants, especially on feasibility and effectiveness. The smaller DeepSeek-32B is competitive overall but less novel: 7.00 under zero-shot versus 7.61 for GPT-4o with 5-shot.
- **Humans vs. LLM judges:** human scores can differ sharply from LLM scores. For GPT-4o with 2-shot prompting, the LLM judge gave 6.84 while human experts gave 4.25.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Farhana Keya, Gollam Rabby, Prasenjit Mitra, Sahar Vahdati, Soren Auer and Yaser Jaradeh. 2025. SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings. arXiv:2503.19257.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{keya2025sci,
  title = {{SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings}},
  author = {Farhana Keya, Gollam Rabby, Prasenjit Mitra, Sahar Vahdati, Soren Auer and Yaser Jaradeh},
  year = {2025},
  doi = {10.48550/arxiv.2503.19257},
  eprint = {2503.19257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.19257}
}
```

- [← previous Referential ambiguity and clarification requests: comparing human and LLM behaviour](https://md.research.elamur.ai/free/en/papers/madge-2025-referential-ambiguity-clarification-requests-comparing/)
- [next → SciClaims: An End-to-End Generative System for Biomedical Claim Analysis](https://md.research.elamur.ai/free/en/papers/ortega-2025-sciclaims-end-end-generative-system/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

7 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774 (2023) 2. Baek, J., Jauhar, S.K., Cucerzan, S., Hwang, S.J.: Researchagent: Iterative research idea gene…
2. The Wharton School Research Paper Forthcoming (2023) 9. Gottweis, J., Weng, W.H., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., et al.: Towards an ai co-scientist. arXiv preprint arXiv:2502.18864 (2025) 10. Gu, X., Krenn, M.: Interesting scient… — [Accelerating scientific discovery with Co-Scientist](https://md.research.elamur.ai/free/en/papers/gottweis-2025-accelerating-scientific-discovery-co/)
3. SCI-IDEA 12. Guo, S., Shariatmadari, A.H., Xiong, G., Huang, A., Xie, E., Bekiranov, S., Zhang, A.: Ideabench: Benchmarking large language models for research idea generation. arXiv preprint arXiv:2411.[02429 (2024) 13](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models ar…
4. Springer Science & Business Media (2013) 15. Li, R., Jing, L., Han, C., Zhou, J., Du, X.: Learning to generate research idea with dynamic control. arXiv preprint arXiv:2412.14626 (2024) 16. Li, S., Padilla, S., Bras, P.L., Dong, J., Chantler, M.: A review of llm-assisted ideation. arXiv preprint arX… [find](https://scholar.google.com/scholar?q=Springer+Science+%26+Business+Media+%282013%29+15%2E+Li%2C+R%2E%2C+Jing%2C+L%2E%2C+Han%2C+C%2E%2C+Zhou%2C+J%2E%2C+Du%2C+X%2E%3A+Learning+to+generate+research+idea+with+dynamic+control%2E+arXiv+preprint+arXiv%3A2412%2E14626+%282024%29+16%2E+Li%2C+S%2E%2C+Pa)
5. OpenAI Blog (2019), — [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/) [https://openai.com/blog/](https://openai.com/blog/) language-unsupervised 26. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. on Natural Language Processing, EMNLP-IJCNLP. [↗](https://openai.com/blog/)
6. ACL (2019). [https://doi.org/10.18653/V1/D19-1410 27](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). Si, C., Yang, D., Hashimoto, T.: Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109 (2024) 28. Srinivasan, G., Lee, C., Sengupta, A., Panda, P., Sarwar, S.S., Roy, K.: Trai… [↗](https://doi.org/10.18653/V1/D19-1410)
7. F. Keya et al. 29. Uhlmann, E.L., Ebersole, C.R., Chartier, C.R., Errington, T.M., Kidwell, M.C., Lai, C.K., McCarthy, R.J., Riegelman, A., Silberzahn, R., Nosek, B.A.: Scientific utopia iii: Crowdsourcing science. Perspectives on Psychological Science 14(5), 711–733 (2019) [find](https://scholar.google.com/scholar?q=F%2E+Keya+et+al%2E+29%2E+Uhlmann%2C+E%2EL%2E%2C+Ebersole%2C+C%2ER%2E%2C+Chartier%2C+C%2ER%2E%2C+Errington%2C+T%2EM%2E%2C+Kidwell%2C+M%2EC%2E%2C+Lai%2C+C%2EK%2E%2C+McCarthy%2C+R%2EJ%2E%2C+Riegelman%2C+A%2E%2C+Silberzahn%2C+R%2E%2C+Nosek%2C+B%2EA%2E%3A+Scientific+utopia+iii%3A+Crowdsou)

*Updated 31 October 2026*