---
title: "Autonomous LLM-driven research from data to human-verifiable"
description: "The authors ask whether an LLM-based system can carry out a full hypothesis-testing study by itself… 49 references, cited by 24 papers here."
authors: ["Tal Ifargan", "Lukas Hafner", "Maor Kern", "Ori Alcalay", "Roy Kishony"]
year: 2024
venue: "arXiv:2404.17605"
source: "https://arxiv.org/abs/2404.17605"
doi: "10.48550/arxiv.2404.17605"
arxiv: "2404.17605"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ifargan-2024-autonomous-llm-driven-research/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ifargan-2024-autonomous-llm-driven-research/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Autonomous LLM-driven research from data to human-verifiable research papers

# Autonomous LLM-driven research from data to human-verifiable research papers

**Authors:** Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay and Roy Kishony · **Year:** 2024 · **Venue:** [arXiv:2404.17605](https://md.research.elamur.ai/free/en/venues/nejm-ai/) ([NEJM AI 2024](https://md.research.elamur.ai/free/en/venues/nejm-ai/2024/)) · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.17605) · [PDF](https://arxiv.org/pdf/2404.17605) · [DOI 10.48550/arxiv.2404.17605](https://doi.org/10.48550/arxiv.2404.17605) · [arXiv 2404.17605](https://arxiv.org/abs/2404.17605) · [OpenAlex](https://openalex.org/W4404941741)

**Citations:** 33 citations (OpenAlex, 2026-10-14) · [24](https://md.research.elamur.ai/free/en/papers/ifargan-2024-autonomous-llm-driven-research/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors ask whether an LLM-based system can carry out a full hypothesis-testing study by itself while staying transparent and checkable. They built data-to-paper, a platform that moves LLM agents and rule-based agents through a fixed sequence of research steps. These steps cover data exploration, goal and hypothesis formulation, analysis planning, code writing and debugging, table creation, literature retrieval and section-by-section writing. Every step runs as its own conversation and receives only selected earlier outputs. Each output passes through rule-based checks and an LLM reviewer, with optional human comments. Because information flow is controlled this way, every number in the final paper can be traced back to the code that produced it. In autopilot mode, given only annotated public datasets, the system wrote complete manuscripts. Their originality was modest, but most were analytically sound. For simple fixed goals, roughly 80–90% of runs reproduced published findings without major errors. Harder goals needed a human co-pilot. The work suggests that automating research can increase verifiability rather than reduce it.

## Key points
- **Workflow:** The pipeline has 17 steps and uses OpenAI ChatGPT models (gpt-3.5-turbo variants and gpt-4). Citations come from the Semantic Scholar Academic Graph API, and the final PDF carries an AI watermark.
- **Open-goal datasets:** Two datasets were used. "Health Indicators" is a BRFSS 2015 subset with 253,680 responses and 22 features. "Social Network" is a Twitter interaction graph of the 117th US Congress.
- **Open-goal results:** Five autopilot runs per dataset (about one hour each) produced 10 papers. Eight had correct analyses with only small wording issues, and two contained fundamental errors.
- **Open-source models:** Llama 2 and CodeLlama hallucinated dataset features and often failed to converge during coding, so they could not complete full research cycles.
- **Availability:** Code is at github.com/Technion-Kishony-lab/data-to-paper.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 59 works · 7,590 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 83 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay and Roy Kishony. 2024. Autonomous LLM-driven research from data to human-verifiable research papers. arXiv:2404.17605.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ifargan2024autonomous,
  title = {{Autonomous LLM-driven research from data to human-verifiable research papers}},
  author = {Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay and Roy Kishony},
  year = {2024},
  doi = {10.48550/arxiv.2404.17605},
  eprint = {2404.17605},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.17605}
}
```

- [← previous A New Massive Multilingual Dataset for High-Performance Language Technologies](https://md.research.elamur.ai/free/en/papers/gibert-2024-new-massive-multilingual-dataset-high/)
- [next → Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, T. Gui, The Rise and Potential of Large Language Mode… — [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
2. Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, Z. Wu, L. Zhao, D. Zhu, X. Li, N. Qiang, D. Shen, T. Liu, B. Ge, [Summary of ChatGPT-Related research and perspective towards the future of large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-summary-chatgpt-related-research-perspective/). Meta-Radiology 1, 100017 (2023).
3. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, J.-R. Wen, [A Survey on Large Language Model based Autonomous Agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/), arXiv [cs.AI] (2023). http://arxiv.org/abs/2308.11432. [↗](http://arxiv.org/abs/2308.11432)
4. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Frei… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
5. G. Spitale, N. Biller-Andorno, F. Germani, AI model GPT-3 (dis)informs us better than humans. Sci Adv 9, eadh1850 (2023). [find](https://scholar.google.com/scholar?q=G%2E+Spitale%2C+N%2E+Biller%2DAndorno%2C+F%2E+Germani%2C+AI+model+GPT%2D3+%28dis%29informs+us+better+than+humans%2E+Sci+Adv+9%2C+eadh1850+%282023%29%2E)
6. J. He-Yueya, G. Poesia, R. E. Wang, N. D. Goodman, [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://md.research.elamur.ai/free/en/papers/yueya-2023-solving-math-word-problems-combining/), arXiv [cs.CL] (2023). http://arxiv.org/abs/2304.09102. [↗](http://arxiv.org/abs/2304.09102)
7. B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, A. Fawzi, [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). Nature 625, 468–475 (2024).
8. T. H. Trinh, Y. Wu, Q. V. Le, H. He, T. Luong, [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature 625, 476–482 (2024).
9. C. Qian, X. Cong, W. Liu, C. Yang, W. Chen, Y. Su, Y. Dang, J. Li, J. Xu, D. Li, Z. Liu, M. Sun, Communicative Agents for Software Development, arXiv [cs.SE] (2023). [http://arxiv.org/abs/2307.07924](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/). [↗](http://arxiv.org/abs/2307.07924)
10. Y. Dong, X. Jiang, Z. Jin, G. Li, [Self-collaboration Code Generation via ChatGPT](https://md.research.elamur.ai/free/en/papers/dong-2023-self-collaboration-code-generation/), arXiv [cs.SE] (2023). http://arxiv.org/abs/2304.07590. [↗](http://arxiv.org/abs/2304.07590)
11. W. Zhou, Y. E. Jiang, P. Cui, T. Wang, Z. Xiao, Y. Hou, R. Cotterell, M. Sachan, [RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text](https://md.research.elamur.ai/free/en/papers/zhou-2023-recurrentgpt-interactive-generation-arbitrarily-long/), arXiv [cs.CL] (2023). http://arxiv.org/abs/2305.13304. [↗](http://arxiv.org/abs/2305.13304)
12. V. Nair, E. Schumacher, G. Tso, A. Kannan, [DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents](https://md.research.elamur.ai/free/en/papers/nair-2023-dera-enhancing-large-language-model/), arXiv [cs.CL] (2023). http://arxiv.org/abs/2303.17071. [↗](http://arxiv.org/abs/2303.17071)
13. T. Tu, A. Palepu, M. Schaekermann, K. Saab, J. Freyberg, R. Tanno, A. Wang, B. Li, M. Amin, N. Tomasev, S. Azizi, K. Singhal, Y. Cheng, L. Hou, A. Webson, K. Kulkarni, S. Sara Mahdavi, C. Semturs, J. Gottweis, J. Barral, K. Chou, G. S. Corrado, Y. Matias, A. Karthikesalingam, V. Natarajan, Towards C… — [arXiv:2401.05654](https://arxiv.org/abs/2401.05654) [find](https://scholar.google.com/scholar?q=T%2E+Tu%2C+A%2E+Palepu%2C+M%2E+Schaekermann%2C+K%2E+Saab%2C+J%2E+Freyberg%2C+R%2E+Tanno%2C+A%2E+Wang%2C+B%2E+Li%2C+M%2E+Amin%2C+N%2E+Tomasev%2C+S%2E+Azizi%2C+K%2E+Singhal%2C+Y%2E+Cheng%2C+L%2E+Hou%2C+A%2E+Webson%2C+K%2E+Kulkarni%2C+S%2E+Sara+Mahdavi%2C+C%2E+Semturs%2C+J%2E+)
14. D. A. Boiko, R. MacKnight, B. Kline, G. Gomes, [Autonomous chemical research with large language models](https://md.research.elamur.ai/free/en/papers/boiko-2023-autonomous-chemical-research-large-language/). Nature 624, 570–578 (2023).
15. A. Birhane, A. Kasirzadeh, D. Leslie, S. Wachter, Science in the age of large language models. Nature Reviews Physics 5, 277–280 (2023). [find](https://scholar.google.com/scholar?q=A%2E+Birhane%2C+A%2E+Kasirzadeh%2C+D%2E+Leslie%2C+S%2E+Wachter%2C+Science+in+the+age+of+large+language+models%2E+Nature+Reviews+Physics+5%2C+277%E2%80%93280+%282023%29%2E)
16. G. Conroy, How ChatGPT and other AI tools could disrupt scientific publishing. Nature 622, 234–236 (2023). [find](https://scholar.google.com/scholar?q=G%2E+Conroy%2C+How+ChatGPT+and+other+AI+tools+could+disrupt+scientific+publishing%2E+Nature+622%2C+234%E2%80%93236+%282023%29%2E)
17. C. Stokel-Walker, R. Van Noorden, What ChatGPT and generative AI mean for science. Nature 614, 214–216 (2023). [find](https://scholar.google.com/scholar?q=C%2E+Stokel%2DWalker%2C+R%2E+Van+Noorden%2C+What+ChatGPT+and+generative+AI+mean+for+science%2E+Nature+614%2C+214%E2%80%93216+%282023%29%2E)
18. C. Stokel-Walker, ChatGPT listed as author on research papers: many scientists disapprove. Nature 613, 620–621 (2023). [find](https://scholar.google.com/scholar?q=C%2E+Stokel%2DWalker%2C+ChatGPT+listed+as+author+on+research+papers%3A+many+scientists+disapprove%2E+Nature+613%2C+620%E2%80%93621+%282023%29%2E)
19. M. Hutson, Could AI help you to write your next paper? Nature 611, 192–193 (2022). [find](https://scholar.google.com/scholar?q=M%2E+Hutson%2C+Could+AI+help+you+to+write+your+next+paper%3F+Nature+611%2C+192%E2%80%93193+%282022%29%2E)
20. V. Berdejo-Espinola, T. Amano, AI tools can improve equity in science. Science 379, 991 (2023). [find](https://scholar.google.com/scholar?q=V%2E+Berdejo%2DEspinola%2C+T%2E+Amano%2C+AI+tools+can+improve+equity+in+science%2E+Science+379%2C+991+%282023%29%2E)
21. L. Messeri, M. J. Crockett, Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024). [find](https://scholar.google.com/scholar?q=L%2E+Messeri%2C+M%2E+J%2E+Crockett%2C+Artificial+intelligence+and+illusions+of+understanding+in+scientific+research%2E+Nature+627%2C+49%E2%80%9358+%282024%29%2E)
22. C. L. Bockting, E. A. M. van Dis, R. van Rooij, W. Zuidema, J. Bollen, Living guidelines for generative AI - why scientists must oversee its use. Nature 622, 693–696 (2023). [find](https://scholar.google.com/scholar?q=C%2E+L%2E+Bockting%2C+E%2E+A%2E+M%2E+van+Dis%2C+R%2E+van+Rooij%2C+W%2E+Zuidema%2C+J%2E+Bollen%2C+Living+guidelines+for+generative+AI+%2D+why+scientists+must+oversee+its+use%2E+Nature+622%2C+693%E2%80%93696+%282023%29%2E)
23. E. B. Wilson, An Introduction to Scientific Research (Courier Corporation, 1990; [https://play.google.com/store/books/details?id=rKCHDQAAQBAJ](https://play.google.com/store/books/details?id=rKCHDQAAQBAJ)). [↗](https://play.google.com/store/books/details?id=rKCHDQAAQBAJ)
24. A. Teboul, Diabetes Health Indicators Dataset (2021). [https://www.kaggle.com/datasets/alexteboul/diabetes-health-indicators-dataset](https://www.kaggle.com/datasets/alexteboul/diabetes-health-indicators-dataset). [↗](https://www.kaggle.com/datasets/alexteboul/diabetes-health-indicators-dataset)
25. C. G. Fink, N. Omodt, S. Zinnecker, G. Sprint, A Congressional Twitter network dataset quantifying pairwise probability of influence. Data Brief 50, 109521 (2023). [find](https://scholar.google.com/scholar?q=C%2E+G%2E+Fink%2C+N%2E+Omodt%2C+S%2E+Zinnecker%2C+G%2E+Sprint%2C+A+Congressional+Twitter+network+dataset+quantifying+pairwise+probability+of+influence%2E+Data+Brief+50%2C+109521+%282023%29%2E)
26. A. L. Saint-Fleur, H. E. Alcalá, S. Sridhar, Outcomes of neonates born through meconium-stained amniotic fluid pre and post 2015 NRP guideline implementation. PLoS One 18, e0289945 (2023). [find](https://scholar.google.com/scholar?q=A%2E+L%2E+Saint%2DFleur%2C+H%2E+E%2E+Alcal%C3%A1%2C+S%2E+Sridhar%2C+Outcomes+of+neonates+born+through+meconium%2Dstained+amniotic+fluid+pre+and+post+2015+NRP+guideline+implementation%2E+PLoS+One+18%2C+e0289945+%282023%29%2E)
27. J.-G. Shim, K.-H. Ryu, S. H. Lee, E.-A. Cho, S. Lee, J. H. Ahn, Machine learning model for predicting the optimal depth of tracheal tube insertion in pediatric patients: A retrospective cohort study. PLoS One 16, e0257069 (2021). [find](https://scholar.google.com/scholar?q=J%2E%2DG%2E+Shim%2C+K%2E%2DH%2E+Ryu%2C+S%2E+H%2E+Lee%2C+E%2E%2DA%2E+Cho%2C+S%2E+Lee%2C+J%2E+H%2E+Ahn%2C+Machine+learning+model+for+predicting+the+optimal+depth+of+tracheal+tube+insertion+in+pediatric+patients%3A+A+retrospective+cohort+study%2E+)
28. R. Kinney, C. Anastasiades, R. Authur, I. Beltagy, J. Bragg, A. Buraczynski, I. Cachola, S. Candra, Y. Chandrasekhar, A. Cohan, M. Crawford, D. Downey, J. Dunkelberger, O. Etzioni, R. Evans, S. Feldman, J. Gorney, D. Graham, F. Hu, R. Huff, D. King, S. Kohlmeier, B. Kuehl, M. Langan, D. Lin, H. Liu,… [find](https://scholar.google.com/scholar?q=R%2E+Kinney%2C+C%2E+Anastasiades%2C+R%2E+Authur%2C+I%2E+Beltagy%2C+J%2E+Bragg%2C+A%2E+Buraczynski%2C+I%2E+Cachola%2C+S%2E+Candra%2C+Y%2E+Chandrasekhar%2C+A%2E+Cohan%2C+M%2E+Crawford%2C+D%2E+Downey%2C+J%2E+Dunkelberger%2C+O%2E+Etzioni%2C+R%2E+Evans%2C+S%2E+Feldma)
29. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, W. Dai, A. Madotto, P. Fung, [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/), arXiv [cs.CL] (2022). http://arxiv.org/abs/2202.03629. [↗](http://arxiv.org/abs/2202.03629)
30. OpenAI platform. [https://platform.openai.com/docs/api-reference](https://platform.openai.com/docs/api-reference). [↗](https://platform.openai.com/docs/api-reference)
31. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
32. B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, G. Synnaeve, Code Llama: Ope… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
33. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, C. Wang, [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/), arXiv [cs.AI] (2023). http://arxiv.org/abs/2308.08155. [↗](http://arxiv.org/abs/2308.08155)
34. S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, J. Schmidhuber, [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/), arXiv [cs.AI] (2023). http://arxiv.org/abs/2308.00352. [↗](http://arxiv.org/abs/2308.00352)
35. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, P. Clark, Self-Refine: Iterative Refinement with Self-Feedback, arXiv [cs.CL] (2023). http://arxiv.org/abs/2303.17651. [↗](http://arxiv.org/abs/2303.17651)
36. C. Harrison, LangChain (2022). [https://github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain). [↗](https://github.com/langchain-ai/langchain)
37. S. Gravitas, AutoGPT (2023). [https://github.com/Significant-Gravitas/AutoGPT](https://github.com/Significant-Gravitas/AutoGPT). [↗](https://github.com/Significant-Gravitas/AutoGPT)
38. B. Peng, M. Galley, P. He, H. Cheng, Y. Xie, Y. Hu, Q. Huang, L. Liden, Z. Yu, W. Chen, J. Gao, [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/), arXiv [cs.CL] (2023). http://arxiv.org/abs/2302.12813. [↗](http://arxiv.org/abs/2302.12813)
39. L. Rolle-Lake, E. Robbins, “Behavioral Risk Factor Surveillance System” in StatPearls (StatPearls Publishing, Treasure Island (FL), 2023; [https://www.ncbi.nlm.nih.gov/pubmed/31971707](https://www.ncbi.nlm.nih.gov/pubmed/31971707)). [↗](https://www.ncbi.nlm.nih.gov/pubmed/31971707)
40. A. Cohan, S. Feldman, I. Beltagy, D. Downey, D. S. Weld, [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://md.research.elamur.ai/free/en/papers/cohan-2020-specter-document-level-representation-learning/), arXiv [cs.CL] (2020). http://arxiv.org/abs/2004.07180. [↗](http://arxiv.org/abs/2004.07180)
41. C. Steinruecken, E. Smith, D. Janz, J. Lloyd, Z. Ghahramani, “The Automatic Statistician” in Automated Machine Learning: Methods, Systems, Challenges, F. Hutter, L. Kotthoff, J. Vanschoren, Eds. (Springer International Publishing, Cham, 2019; https://doi.org/10.1007/978-3-030-05318-5_9), pp. 161–173… — [doi:10.1007/978-3-030-05318-5_9](https://doi.org/10.1007/978-3-030-05318-5_9) [↗](https://doi.org/10.1007/978-3-030-05318-5_9)
42. N. Altman, M. Krzywinski, P values and the search for significance. Nat. Methods 14, 3–4 (2016). [find](https://scholar.google.com/scholar?q=N%2E+Altman%2C+M%2E+Krzywinski%2C+P+values+and+the+search+for+significance%2E+Nat%2E+Methods+14%2C+3%E2%80%934+%282016%29%2E)
43. R. Van Noorden, Hundreds of gibberish papers still lurk in the scientific literature. Nature 594, 160–161 (2021). [find](https://scholar.google.com/scholar?q=R%2E+Van+Noorden%2C+Hundreds+of+gibberish+papers+still+lurk+in+the+scientific+literature%2E+Nature+594%2C+160%E2%80%93161+%282021%29%2E)
44. G. Cabanac, C. Labbé, Prevalence of nonsensical algorithmically generated papers in the scientific literature. J. Assoc. Inf. Sci. Technol. 72, 1461–1476 (2021). [find](https://scholar.google.com/scholar?q=G%2E+Cabanac%2C+C%2E+Labb%C3%A9%2C+Prevalence+of+nonsensical+algorithmically+generated+papers+in+the+scientific+literature%2E+J%2E+Assoc%2E+Inf%2E+Sci%2E+Technol%2E+72%2C+1461%E2%80%931476+%282021%29%2E)
45. L. Liverpool, AI intensifies fight against “paper mills” that churn out fake research. Nature 618, 222–223 (2023). [find](https://scholar.google.com/scholar?q=L%2E+Liverpool%2C+AI+intensifies+fight+against+%E2%80%9Cpaper+mills%E2%80%9D+that+churn+out+fake+research%2E+Nature+618%2C+222%E2%80%93223+%282023%29%2E)
46. Twitter interaction network for the US congress (2023). [https://snap.stanford.edu/data/congress-twitter.html](https://snap.stanford.edu/data/congress-twitter.html). [↗](https://snap.stanford.edu/data/congress-twitter.html)
47. Twitter Handles for Members of the 117th Congress, Food Research & Action Center (2021). [https://frac.org/wp-content/uploads/MOC_Twitter-Handles_117th.pdf](https://frac.org/wp-content/uploads/MOC_Twitter-Handles_117th.pdf). [↗](https://frac.org/wp-content/uploads/MOC_Twitter-Handles_117th.pdf)
48. I. Cachola, K. Lo, A. Cohan, D. S. Weld, [TLDR: Extreme Summarization of Scientific Documents](https://md.research.elamur.ai/free/en/papers/cachola-2020-tldr-extreme-summarization-scientific-documents/), arXiv [cs.CL] (2020). http://arxiv.org/abs/2004.15011. [↗](http://arxiv.org/abs/2004.15011)
49. M. Valenzuela-Escarcega, V. A. Ha, O. Etzioni, Identifying Meaningful Citations. AAAI Workshop: Scholarly Big Data (2015). Acknowledgments We thank Ayelet Baram-Tsabari and Yael Rozenblum for discussions and providing data for initial tests, Ofer Sapir for help in organizing data-to-paper repo, Eric… [find](https://scholar.google.com/scholar?q=M%2E+Valenzuela%2DEscarcega%2C+V%2E+A%2E+Ha%2C+O%2E+Etzioni%2C+Identifying+Meaningful+Citations%2E+AAAI+Workshop%3A+Scholarly+Big+Data+%282015%29%2E+Acknowledgments+We+thank+Ayelet+Baram%2DTsabari+and+Yael+Rozenblum+for+discussi)

*Updated 3 November 2026*