---
title: "PaperBench: Evaluating AI's Ability to Replicate AI Research"
description: "The benchmark offers a way to track autonomous ML R&D ability, which is relevant to frontier-model… 43 references, cited by 81 papers here."
authors: ["Giulio Starace", "Oliver Jaffe", "Dane Sherburn", "James Aung", "Jun Shern Chan", "Leon Maksin", "Rachel Dias", "Evan Mays", "Benjamin Kinsella", "Wyatt Thompson", "Johannes Heidecke", "Amelia Glaese", "Tejal Patwardhan"]
year: 2025
venue: "arXiv:2504.01848"
source: "https://arxiv.org/abs/2504.01848"
doi: "10.48550/arxiv.2504.01848"
arxiv: "2504.01848"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / PaperBench: Evaluating AI's Ability to Replicate AI Research

# PaperBench: Evaluating AI's Ability to Replicate AI Research

**Authors:** [Giulio Starace](https://md.research.elamur.ai/free/en/authors/giulio-starace/), [Oliver Jaffe](https://md.research.elamur.ai/free/en/authors/oliver-jaffe/), [Dane Sherburn](https://md.research.elamur.ai/free/en/authors/dane-sherburn/), [James Aung](https://md.research.elamur.ai/free/en/authors/james-aung/), [Jun Shern Chan](https://md.research.elamur.ai/free/en/authors/jun-shern-chan/), [Leon Maksin](https://md.research.elamur.ai/free/en/authors/leon-maksin/), [Rachel Dias](https://md.research.elamur.ai/free/en/authors/rachel-dias/), [Evan Mays](https://md.research.elamur.ai/free/en/authors/evan-mays/), Benjamin Kinsella, Wyatt Thompson, [Johannes Heidecke](https://md.research.elamur.ai/free/en/authors/johannes-heidecke/), [Amelia Glaese](https://md.research.elamur.ai/free/en/authors/amelia-glaese/) and Tejal Patwardhan · **Year:** 2025 · **Venue:** arXiv:2504.01848 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.01848) · [PDF](https://arxiv.org/pdf/2504.01848) · [DOI 10.48550/arxiv.2504.01848](https://doi.org/10.48550/arxiv.2504.01848) · [arXiv 2504.01848](https://arxiv.org/abs/2504.01848) · [OpenAlex](https://openalex.org/W6948437691)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
PaperBench measures whether AI agents can independently reproduce recent machine learning research. Agents receive 20 Spotlight and Oral papers from ICML 2024, covering 12 topic areas. For each paper, an agent must write a complete codebase from scratch, without access to the authors' original code, and include a reproduce.sh entry script. That script is then rerun on a clean Ubuntu 24.04 machine with an A10 GPU.

Grading uses hierarchical rubrics built together with the original paper authors. Each rubric is a weighted tree whose leaves are binary checks of three kinds: Code Development, Execution and Result Match. Because human grading of a single attempt can take many hours, the authors built an LLM-based judge and validated it against human gold labels in a companion benchmark, JudgeEval.

The strongest agent, Claude 3.5 Sonnet (New) with a simple scaffold, averaged only 21.0%. On a small subset, ML PhDs outperformed the models. The benchmark offers a way to track autonomous ML R&D ability, which is relevant to frontier-model safety frameworks.

## Key points
- The benchmark contains 20 ICML 2024 papers with 8,316 individually gradable rubric requirements. Scores propagate up each rubric tree as weighted averages to produce a Replication Score.
- The best automated judge, o3-mini-high with custom scaffolding, reached an F1 of 0.83 on JudgeEval.
- On a 3-paper subset, the best of three PhD attempts scored 41.4% after 48 hours, compared with 26.6% for o1.
- PaperBench Code-Dev is a lighter variant that checks code only and needs no GPU. On it, o1 scored 43.4%, and grading cost fell from about $66 to about $10 per paper.
- A full o1 IterativeAgent run costs roughly $400 per paper, or about $8,000 for all 20. A monitor disqualified 10 of 646 runs for using blacklisted resources.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

## Cite

Cite as: Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese and Tejal Patwardhan. 2025. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{starace2025paperbench,
  title = {{PaperBench: Evaluating AI's Ability to Replicate AI Research}},
  author = {Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese and Tejal Patwardhan},
  year = {2025},
  doi = {10.48550/arxiv.2504.01848},
  eprint = {2504.01848},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.01848}
}
```

- [← previous Retrieval-Augmented Generation with Graphs (GraphRAG)](https://md.research.elamur.ai/free/en/papers/han-2025-retrieval-augmented-generation-graphs-graphrag/)
- [next → Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Proving](https://md.research.elamur.ai/free/en/papers/lin-2025-goedel-prover-frontier-model-open/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Albergo, M. S., Goldstein, M., Boffi, N. M., Ranganath, R., and Vanden-Eijnden, E. Stochastic Interpolants with Data-Dependent Couplings. June 2024. URL [https://openreview.net/forum?id=FFILRGD0jG](https://openreview.net/forum?id=FFILRGD0jG). [↗](https://openreview.net/forum?id=FFILRGD0jG)
2. However, the capability to autonomously replicate and ex-Anthropic. Responsible Scaling Policy. [https://www.anthropic.com/news/announcing-our-up](https://www.anthropic.com/news/announcing-our-up) dated-responsible-scaling-policy, 2024. Accessed: 2025-01-29. [↗](https://www.anthropic.com/news/announcing-our-up)
3. Cai, C., Ye, Z., Feng, L., Qi, J., and Liu, F. Sample-specific Masks for Visual Reprogramming-based Prompting. June 2024a. URL [https://openreview.net/forum](https://openreview.net/forum) ?id=4sikyurTLX. [↗](https://openreview.net/forum)
4. Harvey Team. Introducing BigLaw Bench, August 2024. URL [https://www.harvey.ai/blog/intro](https://www.harvey.ai/blog/intro) ducing-biglaw-bench. [↗](https://www.harvey.ai/blog/intro)
5. Huang, Q., Vora, J., Liang, P., and Leskovec, J. [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). In Forty-first International Conference on Machine Learning, 2024.
6. Jansen, P., Côté, M.-A., Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. DIS-COVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents. arXiv preprint arXiv:2406.06769, 2024. [find](https://scholar.google.com/scholar?q=Jansen%2C+P%2E%2C+Co%CC%82te%CC%81%2C+M%2E%2DA%2E%2C+Khot%2C+T%2E%2C+Bransom%2C+E%2E%2C+Mishra%2C+B%2E+D%2E%2C+Majumder%2C+B%2E+P%2E%2C+Tafjord%2C+O%2E%2C+and+Clark%2C+P%2E+DIS%2DCOVERYWORLD%3A+A+Virtual+Environment+for+Developing+and+Evaluating+Automated+Scientific+D)
7. Cai, D., Modi, C., Pillaud-Vivien, L., Margossian, C., Gower, R. M., Blei, D., and Saul, L. K. Batch and match: black-box variational inference with a score-based divergence. June 2024b. URL [https://openreview.net/forum?id=bplNmU2ROC](https://openreview.net/forum?id=bplNmU2ROC). [↗](https://openreview.net/forum?id=bplNmU2ROC)
8. Jin, X. and Ren, X. What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement. June 2024. URL [https://openreview.net/for](https://openreview.net/for) um?id=bzNwexOPWm. [↗](https://openreview.net/for)
9. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., et al. [MLE-bench: Evaluating machine learning agents on machine learning engineering](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/). arXiv preprint arXiv:2410.07095, 2024.
10. Jing, L., Huang, Z., Wang, X., Yao, W., Yu, W., Ma, K., Zhang, H., Du, X., and Yu, D. DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? arXiv preprint arXiv:2409.07703, 2024. [find](https://scholar.google.com/scholar?q=Jing%2C+L%2E%2C+Huang%2C+Z%2E%2C+Wang%2C+X%2E%2C+Yao%2C+W%2E%2C+Yu%2C+W%2E%2C+Ma%2C+K%2E%2C+Zhang%2C+H%2E%2C+Du%2C+X%2E%2C+and+Yu%2C+D%2E+DSBench%3A+How+Far+Are+Data+Science+Agents+to+Becoming+Data+Science+Experts%3F+arXiv+preprint+arXiv%3A2409%2E07703%2C+2024%2E)
11. Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., Zhang, Q., Wan, Y., Zhou, P., and Sun, L. [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://md.research.elamur.ai/free/en/papers/chen-2024-mllm-judge-assessing-multimodal-llm/). arXiv preprint arXiv:2402.04788, 2024.
12. Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. RewardBench: Evaluating Reward Models for Language Modeling, 2024. URL https://arxiv.org/abs/2403.13787. [↗](https://arxiv.org/abs/2403.13787)
13. Cheng, Z., Wu, X., Yu, J., Yang, S., Wang, G., and Xing, X. RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation. Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R. A Mechanistic Understanding of Alignment Algorithms: A Case Study on… — [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://md.research.elamur.ai/free/en/papers/lee-2024-mechanistic-understanding-alignment-algorithms-case/)
14. Chiang, C.-H. and yi Lee, H. Can Large Language Models Be an Alternative to Human Evaluations?, 2023. URL https://arxiv.org/abs/2305.01937. [↗](https://arxiv.org/abs/2305.01937)
15. Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test-Time Model Adaptation with Only Forward Passes. June 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) qz1Vx1v9iK. [↗](https://openreview.net/forum?id=)
16. DeepMind. Specification Gaming: The Flip Side of AI Ingenuity, 2024. URL [https://deepmind.google/](https://deepmind.google/) discover/blog/specification- gaming-the- flip- side- of- ai- ingenuity/. Accessed: 10 March 2025. [↗](https://deepmind.google/)
17. OpenAI. Preparedness Framework, December 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Preparedness+Framework%2C+December+2023%2E)
18. Frans, K., Park, S., Abbeel, P., and Levine, S. Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings. June 2024. URL [https://openreview.net/forum?id=a6wCNfIj8E](https://openreview.net/forum?id=a6wCNfIj8E). [↗](https://openreview.net/forum?id=a6wCNfIj8E)
19. Pan, A., Bhatia, K., and Steinhardt, J. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. URL https://arxiv.org/abs/ 2201.03544. [↗](https://arxiv.org/abs/)
20. Fu, J., Ng, S.-K., Jiang, Z., and Liu, P. GPTScore: Evaluate as You Desire. arXiv preprint arXiv:2302.04166, 2023. [find](https://scholar.google.com/scholar?q=Fu%2C+J%2E%2C+Ng%2C+S%2E%2DK%2E%2C+Jiang%2C+Z%2E%2C+and+Liu%2C+P%2E+GPTScore%3A+Evaluate+as+You+Desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2C+2023%2E)
21. Rathore, P., Lei, W., Frangella, Z., Lu, L., and Udell, M. Challenges in Training PINNs: A Loss Landscape Perspective. June 2024. URL [https://openreview.net/forum?id=mJGiFr8jLa](https://openreview.net/forum?id=mJGiFr8jLa). [↗](https://openreview.net/forum?id=mJGiFr8jLa)
22. Gloeckler, M., Deistler, M., Weilbach, C. D., Wood, F., and Macke, J. H. All-in-one simulation-based inference. June 2024. URL [https://openreview.net/forum](https://openreview.net/forum) ?id=DL79HYCFFq. [↗](https://openreview.net/forum)
23. Sanchez, G., Spangher, A., Fan, H., Levi, E., and Biderman, S. Stay on Topic with Classifier-Free Guidance. June 2024. URL [https://openreview.net/forum](https://openreview.net/forum) ?id=RiM3cl9MdK. [↗](https://openreview.net/forum)
24. Google DeepMind. Frontier Safety Framework, May 2024. [find](https://scholar.google.com/scholar?q=Google+DeepMind%2E+Frontier+Safety+Framework%2C+May+2024%2E)
25. Sawada, T., Paleka, D., Havrilla, A., Tadepalli, P., Vidas, P., Kranias, A., Nay, J. J., Gupta, K., and Komatsuzaki, A. [ARB: Advanced Reasoning Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/), 2023. URL https://arxiv.org/abs/2307.13692. [↗](https://arxiv.org/abs/2307.13692)
26. J., and Li, X. [Self-Taught Evaluators](https://md.research.elamur.ai/free/en/papers/wang-2024-self-taught-evaluators/), 2024a. URL https://arxiv.org/abs/2408.02666. [↗](https://arxiv.org/abs/2408.02666)
27. Wang, X., Lin, B., Liu, D., Chen, Y.-C., and Xu, C. Bridging Data Gaps in Diffusion Models with Adversarial Noise-Based Transfer Learning. June 2024b. URL [https://openreview.net/forum?id=PpBs2iL0jv](https://openreview.net/forum?id=PpBs2iL0jv). [↗](https://openreview.net/forum?id=PpBs2iL0jv)
28. Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. June 2024. URL [https://openreview.net/forum?id=WLPhywf1si](https://openreview.net/forum?id=WLPhywf1si). [↗](https://openreview.net/forum?id=WLPhywf1si)
29. Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024. [find](https://scholar.google.com/scholar?q=Wijk%2C+H%2E%2C+Lin%2C+T%2E%2C+Becker%2C+J%2E%2C+Jawhar%2C+S%2E%2C+Parikh%2C+N%2E%2C+Broadley%2C+T%2E%2C+Chan%2C+L%2E%2C+Chen%2C+M%2E%2C+Clymer%2C+J%2E%2C+Dhyani%2C+J%2E%2C+et+al%2E+RE%2DBench%3A+Evaluating+frontier+AI+R%26D+capabilities+of+language+model+agents+again)
30. Sharrock, L., Simons, J., Liu, S., and Beaumont, M. Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models. June 2024. URL [https://openreview.net/forum](https://openreview.net/forum) ?id=8viuf9PdzU. [↗](https://openreview.net/forum)
31. Wolczyk, M., Cupial, B., Ostaszewski, M., Bortkiewicz, M., Zajac, M., Pascanu, R., Kucinski, L., and Milos, P. Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem. June 2024. URL [https://openreview.net/forum?id=53iSXb1m8w](https://openreview.net/forum?id=53iSXb1m8w). [↗](https://openreview.net/forum?id=53iSXb1m8w)
32. Shi, J., Gare, G. R., Tian, J., Chai, S., Lin, Z., Vasudevan, A. B., Feng, D., Ferroni, F., and Kong, S. LCA-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies. June 2024. URL https://op enreview.net/forum?id=HPXRzM9BYZ. [find](https://scholar.google.com/scholar?q=Shi%2C+J%2E%2C+Gare%2C+G%2E+R%2E%2C+Tian%2C+J%2E%2C+Chai%2C+S%2E%2C+Lin%2C+Z%2E%2C+Vasudevan%2C+A%2E+B%2E%2C+Feng%2C+D%2E%2C+Ferroni%2C+F%2E%2C+and+Kong%2C+S%2E+LCA%2Don%2Dthe%2DLine%3A+Benchmarking+Out+of+Distribution+Generalization+with+Class+Taxonomies%2E+June+20)
33. Xia, X., Liu, J., Zhang, S., Wu, Q., Wei, H., and Liu, T. Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints. June 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) yb5xV8LFDq. [↗](https://openreview.net/forum?id=)
34. Si, C., Yang, D., and Hashimoto, T. [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). arXiv preprint arXiv:2409.04109, 2024.
35. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. [ReAct: Synergizing Reasoning and Acting in Language Models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/), March 2023. URL http://arxiv.org/abs/2210.03629. arXiv:2210.03629 [cs]. [↗](http://arxiv.org/abs/2210.03629)
36. Siegel, Z. S., Kapoor, S., Nagdir, N., Stroebl, B., and Narayanan, A. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv preprint arXiv:2409.11363, 2024. [find](https://scholar.google.com/scholar?q=Siegel%2C+Z%2E+S%2E%2C+Kapoor%2C+S%2E%2C+Nagdir%2C+N%2E%2C+Stroebl%2C+B%2E%2C+and+Narayanan%2C+A%2E+CORE%2DBench%3A+Fostering+the+Credibility+of+Published+Research+Through+a+Computational+Reproducibility+Agent+Benchmark%2E+arXiv+preprin)
37. Zhao, B., Hajishirzi, H., and Cao, Q. APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference. June 2024. URL [https://openreview.net/forum?id=sb81Xl50JG](https://openreview.net/forum?id=sb81Xl50JG). [↗](https://openreview.net/forum?id=sb81Xl50JG)
38. Singla, J., Agarwal, A., and Pathak, D. SAPG: Split and Aggregate Policy Gradients. June 2024. URL [https://openreview.net/forum?id=4dOJAfXhNV](https://openreview.net/forum?id=4dOJAfXhNV). [↗](https://openreview.net/forum?id=4dOJAfXhNV)
39. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36: 46595–46623, 2023.
40. Sun, H., Zhuang, Y., Wei, W., Zhang, C., and Dai, B. BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models. Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. Agent-as-a-Judge: Evaluate Agents with Agents. a… [find](https://scholar.google.com/scholar?q=Sun%2C+H%2E%2C+Zhuang%2C+Y%2E%2C+Wei%2C+W%2E%2C+Zhang%2C+C%2E%2C+and+Dai%2C+B%2E+BBox%2DAdapter%3A+Lightweight+Adapting+for+Black%2DBox+Large+Language+Models%2E+Zhuge%2C+M%2E%2C+Zhao%2C+C%2E%2C+Ashley%2C+D%2E%2C+Wang%2C+W%2E%2C+Khizbullin%2C+D%2E%2C+Xiong%2C+Y%2E%2C+Liu%2C+)
41. UK AI Safety Institute. Inspect, 2025. URL [https://inspect.ai-safety-institute.org.uk](https://inspect.ai-safety-institute.org.uk) /agents.html\#sec-basic-agent. van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., and Ward, F. R. AI Sandbagging: Language Models Can Strategically Underperform on Evaluations. arXiv preprint arXiv:2406.07… [↗](https://inspect.ai-safety-institute.org.uk)
42. Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P. Scienceworld: Is Your Agent Smarter than a 5th Grader? arXiv preprint arXiv:2203.07540, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+R%2E%2C+Jansen%2C+P%2E%2C+Co%CC%82te%CC%81%2C+M%2E%2DA%2E%2C+and+Ammanabrolu%2C+P%2E+Scienceworld%3A+Is+Your+Agent+Smarter+than+a+5th+Grader%3F+arXiv+preprint+arXiv%3A2203%2E07540%2C+2022%2E)
43. Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi-Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, A. Future Directions in AI Evaluation [find](https://scholar.google.com/scholar?q=Wang%2C+T%2E%2C+Kulikov%2C+I%2E%2C+Golovneva%2C+O%2E%2C+Yu%2C+P%2E%2C+Yuan%2C+W%2E%2C+Dwivedi%2DYu%2C+J%2E%2C+Pang%2C+R%2E+Y%2E%2C+Fazel%2DZarandi%2C+M%2E%2C+Weston%2C+A%2E+Future+Directions+in+AI+Evaluation)

*Updated 2 November 2026*