---
title: "Auto-Discovery-Bench — paper by Chen et al. (2025)"
description: "The authors present the benchmark as a reproducible, low-confound lower-bound test that… 20 references, cited by 6 papers here."
authors: ["Tingting Chen", "Beibei Lin", "Srinivas Anumasa", "Vedant Shah", "Zifeng Yuan", "Qiran Zou", "Anirudh Goyal", "Dianbo Liu"]
year: 2025
venue: "arXiv:2502.15224"
source: "https://arxiv.org/abs/2502.15224"
doi: "10.48550/arxiv.2502.15224"
arxiv: "2502.15224"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-auto-discovery-bench-diagnosing-structured/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-auto-discovery-bench-diagnosing-structured/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) / Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery

# Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery

**Authors:** Tingting Chen, Beibei Lin, Srinivas Anumasa, [Vedant Shah](https://md.research.elamur.ai/free/en/authors/vedant-shah/), Zifeng Yuan, Qiran Zou, [Anirudh Goyal](https://md.research.elamur.ai/free/en/authors/anirudh-goyal/) and [Dianbo Liu](https://md.research.elamur.ai/free/en/authors/dianbo-liu/) · **Year:** 2025 · **Venue:** arXiv:2502.15224 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2502.15224) · [PDF](https://arxiv.org/pdf/2502.15224) · [DOI 10.48550/arxiv.2502.15224](https://doi.org/10.48550/arxiv.2502.15224) · [arXiv 2502.15224](https://arxiv.org/abs/2502.15224) · [OpenAlex](https://openalex.org/W6891957547)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/chen-2025-auto-discovery-bench-diagnosing-structured/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Agents meant to help with scientific discovery must keep a structured picture of a hidden system accurate across many rounds of experiments and feedback. In realistic environments, failures are hard to diagnose because exploration, domain knowledge, noise and memory all interact. Auto-Discovery-Bench separates out this single skill. It uses deterministic oracles, exact ground truth and clean feedback, while keeping the closed hypothesis–intervention–feedback loop. The benchmark includes three controlled tasks: recovering a directed graph (DAG), inferring undirected relations, and identifying symbolic equations. Models are scored by success rate and by the average number of rounds needed to converge.

Accuracy and efficiency drop as the number of variables, the length of the trajectory and the number of distractors grow. Many models struggle once graphs reach 10 nodes. A separate trajectory-tracking test removes intervention choice and hypothesis generation, yet errors remain. This points to long-range state integration as a key bottleneck. The authors present the benchmark as a reproducible, low-confound lower-bound test that complements realistic discovery environments rather than replacing them.

## Key points
- **Directed graph task:** nodes are framed as "molecules" with discrete states in {0, …, S}. Intervening on a node randomly resets the states of its children, and the agent must recover the adjacency matrix. Scoring treats as equivalent any graphs that produce the same observations, using reachability computed from matrix powers.
- **Models evaluated:** gpt-4o, qwen2.5-72b-instruct, claude-3-5-haiku-20241022 and llama-3.1-70b-instruct.
- **Trajectory length:** in the trajectory-tracking diagnostic, Claude's accuracy falls from 69% at length 3 to almost zero by length 10.
- **Chain-of-thought (CoT) prompting:** CoT helps on short sequences (GPT-4o rises from 13% to 100% at length 5), but the gains shrink on longer inputs.
- **Position effects:** with length fixed at 20 and results averaged over 100 trials, accuracy declines steadily at later positions. Even with CoT, Qwen2.5 stays under 70% at position 19.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 221 works · 39,953 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 60 works · 19,677 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 73 works · 14,704 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works

## Cite

Cite as: Tingting Chen, Beibei Lin, Srinivas Anumasa, Vedant Shah, Zifeng Yuan, Qiran Zou, Anirudh Goyal and Dianbo Liu. 2025. Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery. arXiv:2502.15224.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2025auto,
  title = {{Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery}},
  author = {Tingting Chen, Beibei Lin, Srinivas Anumasa, Vedant Shah, Zifeng Yuan, Qiran Zou, Anirudh Goyal and Dianbo Liu},
  year = {2025},
  doi = {10.48550/arxiv.2502.15224},
  eprint = {2502.15224},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.15224}
}
```

- [← previous ArVoice: A Multi-Speaker Dataset for Arabic Speech Synthesis](https://md.research.elamur.ai/free/en/papers/toyin-2025-arvoice-multi-speaker-dataset-arabic/)
- [next → Autoformalizer with Tool Feedback](https://md.research.elamur.ai/free/en/papers/guo-2025-autoformalizer-tool-feedback/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/)

## References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meta AI. 2024. Llama 3.1: Open foundation and fine-tuned chat models. Model Release Notes. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2024%2E+Llama+3%2E1%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+Model+Release+Notes%2E)
2. Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, and Yoshua Bengio. 2024. Efficient causal graph discovery using large language models. arXiv preprint arXiv:2402.01207. — [arXiv:2402.01207](https://arxiv.org/abs/2402.01207) [find](https://scholar.google.com/scholar?q=Thomas+Jiralerspong%2C+Xiaoyin+Chen%2C+Yash+More%2C+Vedant+Shah%2C+and+Yoshua+Bengio%2E+2024%2E+Efficient+causal+graph+discovery+using+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E01207%2E+%E2%80%94+%5BarXiv%3A2402%2E01207%5D%28)
3. Signe Altmäe, Alberto Sola-Leyva, and Andres Salumets. 2023. Artificial intelligence in scientific writing: a friend or a foe? Reproductive BioMedicine Online, 47(1):3–9. [find](https://scholar.google.com/scholar?q=Signe+Altm%C3%A4e%2C+Alberto+Sola%2DLeyva%2C+and+Andres+Salumets%2E+2023%2E+Artificial+intelligence+in+scientific+writing%3A+a+friend+or+a+foe%3F+Reproductive+BioMedicine+Online%2C+47%281%29%3A3%E2%80%939%2E)
4. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, and 1 others. 2021. [Highly accurate protein structure prediction with alphafold](https://md.research.elamur.ai/free/en/papers/jumper-2021-highly-accurate-protein-structure-prediction/). nature, 596(7873):583–589.
5. Anthropic. 2024. Claude 3.5 haiku model card. Accessed: 2024-10-22. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3%2E5+haiku+model+card%2E+Accessed%3A+2024%2D10%2D22%2E)
6. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 28 others. 2024. Qwen2.5: The next generation of large language models with hybrid sc… [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Yunfei+Chu%2C+Zeyu+Cui%2C+Kai+Dang%2C+Xiaodong+Deng%2C+Yang+Fan%2C+Wenbin+Ge%2C+Yu+Han%2C+Fei+Huang%2C+Binyuan+Hui%2C+Luo+Ji%2C+Mei+Li%2C+Junyang+Lin%2C+Runji+Lin%2C+Dayiheng+Liu%2C+Gao+Liu%2C+Chengqiang+Lu%2C+)
7. Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2023. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050. [find](https://scholar.google.com/scholar?q=Emre+K%C4%B1c%C4%B1man%2C+Robert+Ness%2C+Amit+Sharma%2C+and+Chenhao+Tan%2E+2023%2E+Causal+reasoning+and+large+language+models%3A+Opening+a+new+frontier+for+causality%2E+arXiv+preprint+arXiv%3A2305%2E00050%2E)
8. Yimeng Chen, Piotr Pi˛ekos, Mateusz Ostaszewski, Firas Laakom, and Jürgen Schmidhuber. 2025. Physgym: Benchmarking LLMs in interactive physics discovery with controlled priors. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Yimeng+Chen%2C+Piotr+Pi%CB%9Bekos%2C+Mateusz+Ostaszewski%2C+Firas+Laakom%2C+and+J%C3%BCrgen+Schmidhuber%2E+2025%2E+Physgym%3A+Benchmarking+LLMs+in+interactive+physics+discovery+with+controlled+priors%2E+In+The+Thirty%2Dninth+Ann)
9. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152–1157. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+Mawps%3A+A+math+word+problem+repository%2E+In+Proceedings+of+the+2016+conference+of+the+north+american+chapter+o)
10. Stephanie Long, Tibor Schuster, and Alexandre Piché. 2023. Can large language models build causal graphs? arXiv preprint arXiv:2303.05279. — [arXiv:2303.05279](https://arxiv.org/abs/2303.05279) [find](https://scholar.google.com/scholar?q=Stephanie+Long%2C+Tibor+Schuster%2C+and+Alexandre+Pich%C3%A9%2E+2023%2E+Can+large+language+models+build+causal+graphs%3F+arXiv+preprint+arXiv%3A2303%2E05279%2E+%E2%80%94+%5BarXiv%3A2303%2E05279%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E05279%29)
11. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292.
12. Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. 2023. Scaling deep learning for materials discovery. Nature, 624(7990):80–85. [find](https://scholar.google.com/scholar?q=Amil+Merchant%2C+Simon+Batzner%2C+Samuel+S+Schoenholz%2C+Muratahan+Aykol%2C+Gowoon+Cheon%2C+and+Ekin+Dogus+Cubuk%2E+2023%2E+Scaling+deep+learning+for+materials+discovery%2E+Nature%2C+624%287990%29%3A80%E2%80%9385%2E)
13. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2021. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772. [find](https://scholar.google.com/scholar?q=Shen%2DYun+Miao%2C+Chao%2DChun+Liang%2C+and+Keh%2DYih+Su%2E+2021%2E+A+diverse+corpus+for+evaluating+and+developing+english+math+word+problem+solvers%2E+arXiv+preprint+arXiv%3A2106%2E15772%2E)
14. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
15. OpenAI. 2024. Gpt-4o system architecture. Technical Overview. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Gpt%2D4o+system+architecture%2E+Technical+Overview%2E)
16. OpenRouter. 2025. Openrouter: A unified interface for llms. [https://openrouter.ai](https://openrouter.ai). Accessed: 2025-02-16. [↗](https://openrouter.ai)
17. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) arXiv preprint arXiv:2103.07191.
18. Edward O Pyzer-Knapp, Jed W Pitera, Peter WJ Staar, Seiji Takeda, Teodoro Laino, Daniel P Sanders, James Sexton, John R Smith, and Alessandro Curioni. 2022. Accelerating materials discovery using artificial intelligence, high performance computing and robotics. npj Computational Materials, 8(1):84. [find](https://scholar.google.com/scholar?q=Edward+O+Pyzer%2DKnapp%2C+Jed+W+Pitera%2C+Peter+WJ+Staar%2C+Seiji+Takeda%2C+Teodoro+Laino%2C+Daniel+P+Sanders%2C+James+Sexton%2C+John+R+Smith%2C+and+Alessandro+Curioni%2E+2022%2E+Accelerating+materials+discovery+using+arti)
19. P Rajpurkar. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250.
20. Alex Wang. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461. xAI. 2025. Grok 4 model card. Accessed: 2025-08-25.

*Updated 3 November 2026*