--- title: "Towards Execution-Grounded Automated AI Research — Si 2026" description: "LLMs can propose research ideas that sound convincing but fail when tested. 1 reference, cited by 7 papers here." authors: ["Chenglei Si", "Zitong Yang", "Yejin Choi", "Emmanuel Candes", "Diyi Yang", "Tatsunori Hashimoto"] year: 2026 venue: "arXiv:2601.14525" source: "https://arxiv.org/abs/2601.14525" doi: "10.48550/arxiv.2601.14525" arxiv: "2601.14525" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/si-2026-towards-execution-grounded-automated-ai/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/si-2026-towards-execution-grounded-automated-ai/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / Towards Execution-Grounded Automated AI Research # Towards Execution-Grounded Automated AI Research **Authors:** [Chenglei Si](https://md.research.elamur.ai/free/en/authors/chenglei-si/), [Zitong Yang](https://md.research.elamur.ai/free/en/authors/zitong-yang/), [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/), [Emmanuel Candes](https://md.research.elamur.ai/free/en/authors/emmanuel-j-candes/), [Diyi Yang](https://md.research.elamur.ai/free/en/authors/diyi-yang/) and [Tatsunori Hashimoto](https://md.research.elamur.ai/free/en/authors/tatsunori-hashimoto/) · **Year:** 2026 · **Venue:** arXiv:2601.14525 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2601.14525) · [PDF](https://arxiv.org/pdf/2601.14525) · [DOI 10.48550/arxiv.2601.14525](https://doi.org/10.48550/arxiv.2601.14525) · [arXiv 2601.14525](https://arxiv.org/abs/2601.14525) · [OpenAlex](https://openalex.org/W7125390180) **Citations:** 0 citations (OpenAlex, 2026-10-14) · 7 in our library, of the 149,439 papers whose bibliographies we read ## Summary LLMs can propose research ideas that sound convincing but fail when tested. This paper asks two questions: can idea testing be automated at scale, and can models improve from the measured results? The authors built an automated executor. An Implementer turns natural-language ideas into code diffs, a Scheduler assigns GPU resources, and Workers run the jobs and log the metrics. They tested it on two open-ended environments: speeding up nanoGPT pre-training and improving GRPO post-training for math reasoning. Most ideas from frontier LLMs could be implemented and run automatically. Using execution results to drive evolutionary search worked well with few samples. Within ten epochs it beat both baselines, and on post-training it slightly edged out the best human solution. RL from execution reward had a different outcome. It raised the ideator's average score but not its best score, because the model collapsed onto a few simple ideas. The work shows that execution-grounded automated research is feasible and identifies its current limits. ## Key points - **Environments:** - *nanoGPT:* train a 124M GPT-2 on FineWeb to 3.28 validation loss on 8 H100s, measured as wall-clock time. - *GRPO:* finetune Qwen2.5-Math-1.5B on MATH, measured as validation accuracy. - Both include safeguards against reward hacking, such as frozen evaluation code. - **Execution rates:** over 90% of ideas ran successfully in the pre-training environment with Claude-4.5-Sonnet and Claude-4.5-Opus. - **Evolutionary search results:** - Post-training accuracy reached 69.4% against the 48.0% GRPO baseline, compared with 68.8% for the best human expert. - Pre-training time dropped to 19.7 min from the 35.9 min nanoGPT baseline, compared with 2.1 min for the best human expert. - Search beat best-of-N under the same sampling budget. - **Scaling:** only Claude-4.5-Opus showed a clear scaling trend; Claude-4.5-Sonnet and GPT-5 plateaued early. - **RL with Qwen3-30B:** mean reward rose but max reward did not, and thinking length and idea diversity collapsed. Environments and trajectories are released in the GitHub repository NoviScl/Automated-AI-Researcher. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works ## Cite Cite as: Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candes, Diyi Yang and Tatsunori Hashimoto. 2026. Towards Execution-Grounded Automated AI Research. arXiv:2601.14525. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{si2026execution, title = {{Towards Execution-Grounded Automated AI Research}}, author = {Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candes, Diyi Yang and Tatsunori Hashimoto}, year = {2026}, doi = {10.48550/arxiv.2601.14525}, eprint = {2601.14525}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2601.14525} } ``` - [← previous This One or That One? A Study on Accessibility via Demonstratives with Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2026-this-one-or-that-one/) - [next → TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization](https://md.research.elamur.ai/free/en/papers/ma-2026-tspo-breaking-double-homogenization-dilemma/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Allen-Zhu, Z. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. ArXiv, 2025. [find](https://scholar.google.com/scholar?q=Allen%2DZhu%2C+Z%2E+Physics+of+Language+Models%3A+Part+4%2E1%2C+Architecture+Design+and+the+Magic+of+Canon+Layers%2E+ArXiv%2C+2025%2E) *Updated 30 October 2026*