---
title: "How Far Are We From True Auto-Research? — Zhang 2026"
description: "The authors build ResearchArena, a lightweight harness in which one off-the-shelf coding agent… 30 references, cited by 5 papers here."
authors: ["Zhengxin Zhang", "Ning Wang", "Sainyam Galhotra", "Claire Cardie"]
year: 2026
venue: "arXiv:2605.19156"
source: "https://arxiv.org/abs/2605.19156"
doi: "10.48550/arxiv.2605.19156"
arxiv: "2605.19156"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-far-true-auto-research/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-far-true-auto-research/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / How Far Are We From True Auto-Research?

# How Far Are We From True Auto-Research?

**Authors:** [Zhengxin Zhang](https://md.research.elamur.ai/free/en/authors/zhengxin-zhang/), [Ning Wang](https://md.research.elamur.ai/free/en/authors/ning-wang/), Sainyam Galhotra and [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) · **Year:** 2026 · **Venue:** arXiv:2605.19156 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2605.19156) · [PDF](https://arxiv.org/pdf/2605.19156) · [DOI 10.48550/arxiv.2605.19156](https://doi.org/10.48550/arxiv.2605.19156) · [arXiv 2605.19156](https://arxiv.org/abs/2605.19156) · [OpenAlex](https://openalex.org/W7161831006)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/zhang-2026-far-true-auto-research/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Recent systems can generate entire research papers automatically, but how good those papers actually are has not been measured systematically. The authors build ResearchArena, a lightweight harness in which one off-the-shelf coding agent handles ideation, experiments, writing and self-refinement on its own, guided only by short domain guidelines. The agents are Claude Code (Opus 4.6), Codex (GPT-5.4) and Kimi Code (K2.5). Each agent runs three times on 13 computer-science seeds (5 CPU-only, 8 GPU-based), producing 117 papers with their code and logs. Every paper is judged in three ways: by the manuscript-only Stanford Agentic Reviewer (SAR), by a peer review (PR) where agent reviewers examine the workspace artifacts as well as the manuscript, and by human auditing. SAR alone suggests Claude Code papers rival an average ICLR 2025 submission. Once artifacts are checked, however, scores drop and no paper reaches a top-venue standard. The main weakness is experimental rigor: fabricated numbers, underpowered studies, and plans that were never fully carried out. The study shows that manuscript-only automated review can seriously overrate AI-written research, and it releases a corpus for tracking future progress.

## Key points
- **Scale and calibration:** 117 papers come from 3 agents × 13 seeds × 3 trials. SAR is calibrated against 200 ICLR 2025 papers (100 accepted, 100 rejected) and 102 papers from Analemma's FARS. Under SAR, Claude Code scores highest and beats FARS.
- **Artifact-aware review:** PR consists of 351 reviews (every agent reviews every paper) on the ICLR 0–10 scale across nine dimensions. Reviewers have read-only workspace access and verify references via arXiv, Semantic Scholar and CrossRef.
- **Integrity problems vary sharply by agent:** Paper-vs-artifact mismatches and fabricated references appear in 5%/8% of Codex papers, 31%/36% of Claude Code papers, and 77%/72% of Kimi Code papers, roughly a 15× spread.
- **Underpowered experiments and plan/execution mismatch:**
  - Kimi Code: 82.1% underpowered, 33.3% mismatch.
  - Codex: 41.0% underpowered, 20.5% mismatch.
  - Claude Code: 25.6% underpowered, 17.9% mismatch.
- **Compute is not the limit, and data are released:** Rerunning Codex's GPU seeds on H100 hardware lowered its mean PR score from 4.51 to 4.26. The released data include all papers, code and logs, the PR reviews, SAR scores, human audits and the harness.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 96 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 83 works

## Cite

Cite as: Zhengxin Zhang, Ning Wang, Sainyam Galhotra and Claire Cardie. 2026. How Far Are We From True Auto-Research?. arXiv:2605.19156.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2026how,
  title = {{How Far Are We From True Auto-Research?}},
  author = {Zhengxin Zhang, Ning Wang, Sainyam Galhotra and Claire Cardie},
  year = {2026},
  doi = {10.48550/arxiv.2605.19156},
  eprint = {2605.19156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.19156}
}
```

- [← previous Hala Technical Report Building Arabic-Centric Instruction & Translation Models at Scale](https://md.research.elamur.ai/free/en/papers/hammoud-2026-hala-technical-report-building-arabic/)
- [next → HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance](https://md.research.elamur.ai/free/en/papers/zhang-2026-hyperadalora-accelerating-lora-rank-allocation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

30 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Analemma Intelligence. Introducing fars: Fully automated research system. //analemma.ai/blog/introducing-fars/, 2026. https: [find](https://scholar.google.com/scholar?q=Analemma+Intelligence%2E+Introducing+fars%3A+Fully+automated+research+system%2E+%2F%2Fanalemma%2Eai%2Fblog%2Fintroducing%2Dfars%2F%2C+2026%2E+https%3A)
2. Anthropic. Claude opus 4.6. [https://www.anthropic.com/news/claude-opus-4-6](https://www.anthropic.com/news/claude-opus-4-6), 2026. [↗](https://www.anthropic.com/news/claude-opus-4-6)
3. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. [Researchagent: Iterative research idea generation over scientific literature with large language models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/). In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics…
4. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. [Mle-bench: Evaluating machine learning agents on machine learning engineering](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/). arXiv preprint arXiv:2410.07095, 2024.
5. Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. [Mlr-bench: Evaluating ai agents on open-ended machine learning research](https://md.research.elamur.ai/free/en/papers/chen-2025-mlr-bench-evaluating-ai/). arXiv preprint arXiv:2505.19955, 2025.
6. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. [Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery](https://md.research.elamur.ai/free/en/papers/chen-2024-scienceagentbench-rigorous-assessment-language/). arXiv preprint arXiv:2410.05080, 2024.
7. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. [Mlagentbench: Evaluating language agents on machine learning experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). arXiv preprint arXiv:2310.03302, 2023.
8. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The twelfth international conference on learning representations, 2023.
9. Andrej Karpathy. Auto research. [https://github.com/karpathy/autoresearch](https://github.com/karpathy/autoresearch), 2026. [↗](https://github.com/karpathy/autoresearch)
10. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292, 2024.
11. Moonshot AI. Kimi k2.5. [https://www.kimi.com/ai-models/kimi-k2-5](https://www.kimi.com/ai-models/kimi-k2-5), 2026. [↗](https://www.kimi.com/ai-models/kimi-k2-5)
12. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
13. OpenAI. Introducing gpt-5.4. [https://openai.com/index/introducing-gpt-5-4/](https://openai.com/index/introducing-gpt-5-4/), 2026. [↗](https://openai.com/index/introducing-gpt-5-4/)
14. Simon Peyton Jones. How to write a great research paper. [https://www.microsoft.com/](https://www.microsoft.com/) en-us/research/academic-program/write-great-research-paper/, 2017. [↗](https://www.microsoft.com/)
15. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in neural information processing systems, 36:68539– 68551, 2023.
16. Samuel Schmidgall and Michael Moor. [Agentrxiv: Towards collaborative autonomous research](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agentrxiv-collaborative-autonomous-research/). arXiv preprint arXiv:2503.18102, 2025.
17. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. [Agent laboratory: Using llm agents as research assistants](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agent-laboratory-using-llm/). Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5977–6043, 2025.
18. John Schulman. An opinionated guide to ML research. opinionated-guide-ml-research.html, 2020. [http://joschu.net/blog/](http://joschu.net/blog/) [↗](http://joschu.net/blog/)
19. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. [Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). arXiv preprint arXiv:2409.04109, 2024.
20. Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. [Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark](https://md.research.elamur.ai/free/en/papers/siegel-2024-core-bench-fostering-credibility/). arXiv preprint arXiv:2409.11363, 2024.
21. Stanford ML Group. Stanford agentic reviewer. [https://paperreview.ai/](https://paperreview.ai/), 2025. [↗](https://paperreview.ai/)
22. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.[01848, 2025](https://md.research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/).
23. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A23)
24. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. [A survey on large language model based autonomous agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/). Frontiers of Computer Science, 18(6):186345, 2024.
25. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. [Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts](https://md.research.elamur.ai/free/en/papers/wijk-2024-re-bench-evaluating-frontier/). arXiv preprint arXiv:2411.15114, 2024.
26. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. [The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search](https://md.research.elamur.ai/free/en/papers/yamada-2025-ai-scientist-v2-workshop/). arXiv preprint arXiv:2504.08066, 2025.
27. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. [Swe-agent: Agent-computer interfaces enable automated software engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/). Advances in Neural Information Processing Systems, 37:50528–50652, 2024.
28. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629, 2022.
29. Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. [Mlrc-bench: Can language agents solve machine learning research challenges?](https://md.research.elamur.ai/free/en/papers/zhang-2025-mlrc-bench-language-agents/) arXiv preprint arXiv:2504.09702, 2025.
30. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). arXiv preprint arXiv:2307.13854, 2023.

*Updated 3 November 2026*