---
title: "MLAgentBench — Huang 2023"
description: "MLAgentBench asks whether agents built on large language models can run machine learning experiments on their own: design an approach, write and execute…"
authors: ["Qian Huang", "Jian Vora", "Percy Liang", "Jure Leskovec"]
year: 2023
venue: "arXiv:2310.03302"
source: "https://arxiv.org/abs/2310.03302"
doi: "10.48550/arxiv.2310.03302"
arxiv: "2310.03302"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

# MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

**Authors:** Qian Huang, Jian Vora, [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/), [Jure Leskovec](https://md.research.elamur.ai/free/en/authors/jure-leskovec/) · **Year:** 2023 · **Venue:** arXiv:2310.03302 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.03302) · [PDF](https://arxiv.org/pdf/2310.03302) · [DOI 10.48550/arxiv.2310.03302](https://doi.org/10.48550/arxiv.2310.03302) · [arXiv 2310.03302](https://arxiv.org/abs/2310.03302) · [OpenAlex](https://openalex.org/W4387432648)

**Citations:** 9 citations (openalex-snapshot · 2026-10-14) · [128](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
MLAgentBench asks whether agents built on large language models can run machine learning experiments on their own: design an approach, write and execute code, read the results, and iterate. The authors built a benchmark of 13 tasks. Each task comes with a goal description, starter files (data and usually a baseline script), and an automatic evaluator that scores the agent's final submission. The tasks range from classic problems such as CIFAR-10 to recent Kaggle competitions, research problems like BabyLM, and code-speedup challenges. Agents work in a sandboxed workspace using file and execution actions. They count as successful if they beat the baseline metric by at least 10%. The authors also designed a ReAct-style agent that, at each step, writes a reflection, a plan, a fact check and its reasoning before choosing an action. They tested it with several LLM backends. Claude v3 Opus performed best, but results depend heavily on how new the task is, with near-total failure on recent challenges. This shows that autonomous ML research remains unreliable, mainly because of weak long-horizon planning and hallucination.

## Key points
- **Task suite:** 13 tasks in five groups:
  - canonical: CIFAR-10, imdb, ogbn-arxiv;
  - classic Kaggle: house-price, spaceship-titanic;
  - recent Kaggle: parkinsons-disease, fathomnet, feedback, identify-contrails;
  - research: CLRS, BabyLM;
  - code speedup: llama-inference, vectorization.
- **Action space:** the environment offers primitive actions (list, read, write, copy and execute files, undo edits) and compound LM-driven actions (Understand File, Edit Script, Edit Script Segment).
- **Models compared:** GPT-4, GPT-4-turbo, Claude v1.0, v2.1 and v3 Opus, Gemini-Pro, and Mixtral.
  - Claude v3 Opus reached a 37.5% average success rate.
  - The best agents succeeded in 100% of 8 runs on house-price but only 0–25% on recent Kaggle tasks and BabyLM.
- **Efficiency:** GPT-4-turbo used about 51% fewer tokens than the average agent. A full benchmark pass with it took roughly 6M tokens, about $60, at around 26% average success.
- **Availability:** code is released at github.com/snap-stanford/MLAgentBench.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 121 not yet.

- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agent-laboratory-using-llm/)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/)
- [Large Language Models for Information Retrieval: A Survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{huang2023mlagentbench,
  title = {{MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation}},
  author = {Huang, Qian and Vora, Jian and Liang, Percy and Leskovec, Jure},
  year = {2023},
  doi = {10.48550/arxiv.2310.03302},
  eprint = {2310.03302},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03302}
}
```

- [← previous Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://md.research.elamur.ai/free/en/papers/leng-2023-mitigating-object-hallucinations-large-vision/)
- [next → MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anna Montoya, D. House prices - advanced regression techniques, 2016. URL [https://kaggle.com/](https://kaggle.com/) competitions/house-prices-advanced-regression-techniques. [↗](https://kaggle.com/)
2. Anthropic. Introducing claude, 2023. [https://www.anthropic.com/index/](https://www.anthropic.com/index/) introducing-claude. [↗](https://www.anthropic.com/index/)
3. URL Berens, P., Cranmer, K., Lawrence, N. D., von Luxburg, U., and Montgomery, J. Ai for science: An emerging agenda. ArXiv, abs/2303.04217, 2023. — [arXiv:2303.04217](https://arxiv.org/abs/2303.04217) [find](https://scholar.google.com/scholar?q=URL+Berens%2C+P%2E%2C+Cranmer%2C+K%2E%2C+Lawrence%2C+N%2E+D%2E%2C+von+Luxburg%2C+U%2E%2C+and+Montgomery%2C+J%2E+Ai+for+science%3A+An+emerging+agenda%2E+ArXiv%2C+abs%2F2303%2E04217%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E04217%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E04217)
4. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL [http://github.com/google/jax](http://github.com/google/jax). [↗](http://github.com/google/jax)
5. Chollet, F. et al. Keras, 2015. URL [https://github.com/fchollet/keras](https://github.com/fchollet/keras). [↗](https://github.com/fchollet/keras)
6. Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: a survey. J. Mach. Learn. Res., 20(1):1997–2017, jan 2019. ISSN 1532-4435. [find](https://scholar.google.com/scholar?q=Elsken%2C+T%2E%2C+Metzen%2C+J%2E+H%2E%2C+and+Hutter%2C+F%2E+Neural+architecture+search%3A+a+survey%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+20%281%29%3A1997%E2%80%932017%2C+jan+2019%2E+ISSN+1532%2D4435%2E)
7. Franklin, A., Maggie, Benner, M., Rambis, N., Baffour, P., Holbrook, R., Crossley, S., and ulrichboser. Feedback prize - english language learning, 2022. URL [https://kaggle.com/competitions/feedback-prize-english-language-learning](https://kaggle.com/competitions/feedback-prize-english-language-learning). [↗](https://kaggle.com/competitions/feedback-prize-english-language-learning)
8. H., Burget, J., Ho, A., Barnes, E., and Christiano, P. F. Evaluating language-model agents on realistic autonomous tasks. ArXiv, abs/2312.11671, 2023. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:260472392. — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) [↗](https://api.semanticscholar.org/)
9. He, X., Zhao, K., and Chu, X. Automl: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622, 2021. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2020.106622. URL https://www.sciencedirect.com/science/ article/pii/S0950705120307516. — [doi:10.1016/j.knosys.2020.106622](https://doi.org/10.1016/j.knosys.2020.106622) [↗](https://doi.org/10.1016/j.knosys.2020.106622)
10. Kirsch, L., Dane, S., Adam, S., and Dardov, V. Amp®-parkinson’s disease progression prediction, 2023. URL [https://kaggle.com/competitions/](https://kaggle.com/competitions/) amp-parkinsons-disease-progression-prediction. [↗](https://kaggle.com/competitions/)
11. Kitano, H. Nobel turing challenge: creating the engine for scientific discovery. NPJ Systems Biology and Applications, 7, 2021. [find](https://scholar.google.com/scholar?q=Kitano%2C+H%2E+Nobel+turing+challenge%3A+creating+the+engine+for+scientific+discovery%2E+NPJ+Systems+Biology+and+Applications%2C+7%2C+2021%2E)
12. Howard, A., Chow, A., and Holbrook, R. Spaceship titanic, 2022. URL [https://kaggle.com/](https://kaggle.com/) competitions/spaceship-titanic. [↗](https://kaggle.com/)
13. Kramer, S., Cerrato, M., Dzeroski, S., and King, R. D. Automated scientific discovery: From equation discovery to autonomous discovery systems. ArXiv, abs/2305.02251, 2023. [find](https://scholar.google.com/scholar?q=Kramer%2C+S%2E%2C+Cerrato%2C+M%2E%2C+Dzeroski%2C+S%2E%2C+and+King%2C+R%2E+D%2E+Automated+scientific+discovery%3A+From+equation+discovery+to+autonomous+discovery+systems%2E+ArXiv%2C+abs%2F2305%2E02251%2C+2023%2E)
14. Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. ArXiv, abs/2005.00687, 2020. [find](https://scholar.google.com/scholar?q=Hu%2C+W%2E%2C+Fey%2C+M%2E%2C+Zitnik%2C+M%2E%2C+Dong%2C+Y%2E%2C+Ren%2C+H%2E%2C+Liu%2C+B%2E%2C+Catasta%2C+M%2E%2C+and+Leskovec%2C+J%2E+Open+graph+benchmark%3A+Datasets+for+machine+learning+on+graphs%2E+ArXiv%2C+abs%2F2005%2E00687%2C+2020%2E)
15. Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. [find](https://scholar.google.com/scholar?q=Krizhevsky%2C+A%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+2009%2E)
16. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+Las+Casas%2C+D%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Bour%2C+G%2E%2C+Lample%2C+G%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Saulnier%2C)
17. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learn… [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Yu%2C+H%2E%2C+Zhang%2C+H%2E%2C+Xu%2C+Y%2E%2C+Lei%2C+X%2E%2C+Lai%2C+H%2E%2C+Gu%2C+Y%2E%2C+Ding%2C+H%2E%2C+Men%2C+K%2E%2C+Yang%2C+K%2E%2C+Zhang%2C+S%2E%2C+Deng%2C+X%2E%2C+Zeng%2C+A%2E%2C+Du%2C+Z%2E%2C+Zhang%2C+C%2E%2C+Shen%2C+S%2E%2C+Zhang%2C+T%2E%2C+Su%2C+Y%2E%2C+Sun%2C+H%2E%2C+Huang%2C+M%2E%2C+Dong%2C+Y%2E%2C+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/)
18. Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, June 2011. Association… [find](https://scholar.google.com/scholar?q=Maas%2C+A%2E+L%2E%2C+Daly%2C+R%2E+E%2E%2C+Pham%2C+P%2E+T%2E%2C+Huang%2C+D%2E%2C+Ng%2C+A%2E+Y%2E%2C+and+Potts%2C+C%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/)
19. Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zı́dek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D. A., C… [find](https://scholar.google.com/scholar?q=Jumper%2C+J%2E+M%2E%2C+Evans%2C+R%2E%2C+Pritzel%2C+A%2E%2C+Green%2C+T%2E%2C+Figurnov%2C+M%2E%2C+Ronneberger%2C+O%2E%2C+Tunyasuvunakool%2C+K%2E%2C+Bates%2C+R%2E%2C+Z%C4%B1%CC%81dek%2C+A%2E%2C+Potapenko%2C+A%2E%2C+Bridgland%2C+A%2E%2C+Meyer%2C+C%2E%2C+Kohl%2C+S%2E+A%2E+A%2E%2C+Ballard%2C+A%2E%2C+Cowie) [our card](https://md.research.elamur.ai/free/en/papers/jumper-2021-highly-accurate-protein-structure-prediction/)
20. Nakano, R., Hilton, J., Balaji, S., Wu, J., Long, O., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332,… [find](https://scholar.google.com/scholar?q=Nakano%2C+R%2E%2C+Hilton%2C+J%2E%2C+Balaji%2C+S%2E%2C+Wu%2C+J%2E%2C+Long%2C+O%2E%2C+Kim%2C+C%2E%2C+Hesse%2C+C%2E%2C+Jain%2C+S%2E%2C+Kosaraju%2C+V%2E%2C+Saunders%2C+W%2E%2C+Jiang%2C+X%2E%2C+Cobbe%2C+K%2E%2C+Eloundou%2C+T%2E%2C+Krueger%2C+G%2E%2C+Button%2C+K%2E%2C+Knight%2C+M%2E%2C+Chess%2C+B%2E%2C+and+) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
21. King, R. D., Whelan, K. E., Jones, F. M., Reiser, P. G. K., Bryant, C. H., Muggleton, S. H., Kell, D. B., and Oliver, S. G. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427:247– 252, 2004. [find](https://scholar.google.com/scholar?q=King%2C+R%2E+D%2E%2C+Whelan%2C+K%2E+E%2E%2C+Jones%2C+F%2E+M%2E%2C+Reiser%2C+P%2E+G%2E+K%2E%2C+Bryant%2C+C%2E+H%2E%2C+Muggleton%2C+S%2E+H%2E%2C+Kell%2C+D%2E+B%2E%2C+and+Oliver%2C+S%2E+G%2E+Functional+genomic+hypothesis+generation+and+experimentation+by+a+robot+scie)
22. King, R. D., Rowland, J. J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L. N., Sparkes, A., Whelan, K. E., and Clare, A. The automation of science. Science, 324:85 – 89, 2009. [find](https://scholar.google.com/scholar?q=King%2C+R%2E+D%2E%2C+Rowland%2C+J%2E+J%2E%2C+Oliver%2C+S%2E+G%2E%2C+Young%2C+M%2E%2C+Aubrey%2C+W%2E%2C+Byrne%2C+E%2E%2C+Liakata%2C+M%2E%2C+Markham%2C+M%2E%2C+Pir%2C+P%2E%2C+Soldatova%2C+L%2E+N%2E%2C+Sparkes%2C+A%2E%2C+Whelan%2C+K%2E+E%2E%2C+and+Clare%2C+A%2E+The+automation+of+science%2E+)
23. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2E+ArXiv%2C+abs%2F2303%2E08774%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
24. Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Mi… [find](https://scholar.google.com/scholar?q=Park%2C+J%2E+S%2E%2C+O%E2%80%99Brien%2C+J%2E%2C+Cai%2C+C%2E+J%2E%2C+Morris%2C+M%2E+R%2E%2C+Liang%2C+P%2E%2C+and+Bernstein%2C+M%2E+S%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36th+Annual+ACM+Symposium+on+User)
25. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:249210177. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L. J., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. ArXiv, abs/2305.16291, 2023a. [↗](https://api.semanticscholar.org/)
26. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-perfor… [find](https://scholar.google.com/scholar?q=Paszke%2C+A%2E%2C+Gross%2C+S%2E%2C+Massa%2C+F%2E%2C+Lerer%2C+A%2E%2C+Bradbury%2C+J%2E%2C+Chanan%2C+G%2E%2C+Killeen%2C+T%2E%2C+Lin%2C+Z%2E%2C+Gimelshein%2C+N%2E%2C+Antiga%2C+L%2E%2C+Desmaison%2C+A%2E%2C+Kopf%2C+A%2E%2C+Yang%2C+E%2E%2C+DeVito%2C+Z%2E%2C+Raison%2C+M%2E%2C+Tejani%2C+A%2E%2C+Chilamku) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
27. Wang, Q., Downey, D., Ji, H., and Hope, T. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259, 2023b. [find](https://scholar.google.com/scholar?q=Wang%2C+Q%2E%2C+Downey%2C+D%2E%2C+Ji%2C+H%2E%2C+and+Hope%2C+T%2E+Scimon%3A+Scientific+inspiration+machines+optimized+for+novelty%2E+arXiv+preprint+arXiv%3A2305%2E14259%2C+2023b%2E)
28. Warstadt, A., Choshen, L., Mueller, A., Williams, A., Wilcox, E. G., and Zhuang, C. Call for papers - the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. ArXiv, abs/2301.11796, 2023. [find](https://scholar.google.com/scholar?q=Warstadt%2C+A%2E%2C+Choshen%2C+L%2E%2C+Mueller%2C+A%2E%2C+Williams%2C+A%2E%2C+Wilcox%2C+E%2E+G%2E%2C+and+Zhuang%2C+C%2E+Call+for+papers+%2D+the+babylm+challenge%3A+Sample%2Defficient+pretraining+on+a+developmentally+plausible+corpus%2E+ArXiv%2C+a)
29. Sarna, A., Elkin, C., inversion, Ng, J., Maggie, and Reade, W. Google research - identify contrails to reduce global warming, 2023. URL [https://kaggle.com/competitions/](https://kaggle.com/competitions/) google-research-identify-contrails-reduce-global-warming. [↗](https://kaggle.com/competitions/)
30. Woodward, B., eor123, GenevievePatterson, and Carlsen, L. Fathomnet 2023, 2023. URL [https://kaggle.com/competitions/](https://kaggle.com/competitions/) fathomnet-out-of-sample-detection. [↗](https://kaggle.com/competitions/)
31. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [find](https://scholar.google.com/scholar?q=Yao%2C+S%2E%2C+Zhao%2C+J%2E%2C+Yu%2C+D%2E%2C+Du%2C+N%2E%2C+Shafran%2C+I%2E%2C+Narasimhan%2C+K%2E%2C+and+Cao%2C+Y%2E+ReAct%3A+Synergizing+reasoning+and+acting+in+language+models%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2C+) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
32. Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/](https://openreview.net/) forum?id… [↗](https://openreview.net/) [our card](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
33. Zhang, L., Zhang, Y., Ren, K., Li, D., and Yang, Y. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. ArXiv, abs/2304.14979, 2023a. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 258418182. [↗](https://api.semanticscholar.org/CorpusID:)
34. Schwaller, P., Gaudin, T., Lanyi, D., Bekas, C., and Laino, T. “found in translation”: predicting outcomes of complex organic chemistry reactions using neural sequence-to-sequence models† †electronic supplementary information (esi) available: Time-split test set and example predictions, together wit… [find](https://scholar.google.com/scholar?q=Schwaller%2C+P%2E%2C+Gaudin%2C+T%2E%2C+Lanyi%2C+D%2E%2C+Bekas%2C+C%2E%2C+and+Laino%2C+T%2E+%E2%80%9Cfound+in+translation%E2%80%9D%3A+predicting+outcomes+of+complex+organic+chemistry+reactions+using+neural+sequence%2Dto%2Dsequence+models%E2%80%A0+%E2%80%A0electronic+)
35. Zhang, M., Qamar, M., Kang, T., Jung, Y., Zhang, C., Bae, S.-H., and Zhang, C. A survey on graph diffusion models: Generative ai in science for molecule, protein and material. ArXiv, abs/2304.01565, 2023b. — [arXiv:2304.01565](https://arxiv.org/abs/2304.01565) [find](https://scholar.google.com/scholar?q=Zhang%2C+M%2E%2C+Qamar%2C+M%2E%2C+Kang%2C+T%2E%2C+Jung%2C+Y%2E%2C+Zhang%2C+C%2E%2C+Bae%2C+S%2E%2DH%2E%2C+and+Zhang%2C+C%2E+A+survey+on+graph+diffusion+models%3A+Generative+ai+in+science+for+molecule%2C+protein+and+material%2E+ArXiv%2C+abs%2F2304%2E01565%2C+2)
36. Zhang, S., Gong, C., Wu, L., Liu, X., and Zhou, M. Automl-gpt: Automatic machine learning with gpt. ArXiv, abs/2305.02499, 2023c. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 258480269. [↗](https://api.semanticscholar.org/CorpusID:)
37. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=vAElhFcKW6](https://openreview.net/forum?id=vAElhFcKW6). [↗](https://openreview.net/forum?id=vAElhFcKW6) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
38. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., and Neubig, G. Webarena: A realistic web environment for building autonomous agents. ArXiv, abs/2307.13854, 2023. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:260164780. [↗](https://api.semanticscholar.org/) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/)
39. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozie%CC%80re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+Rodriguez%2C+A%2E%2C+Joulin%2C+A%2E%2C+Grave%2C+E%2E%2C+and+Lample%2C+G%2E+Llama%3A+Open+and+ef)
40. Velivckovi’c, P., Badia, A. P., Budden, D., Pascanu, R., Banino, A., Dashevskiy, M., Hadsell, R., and Blundell, C. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, 2022. [find](https://scholar.google.com/scholar?q=Velivckovi%E2%80%99c%2C+P%2E%2C+Badia%2C+A%2E+P%2E%2C+Budden%2C+D%2E%2C+Pascanu%2C+R%2E%2C+Banino%2C+A%2E%2C+Dashevskiy%2C+M%2E%2C+Hadsell%2C+R%2E%2C+and+Blundell%2C+C%2E+The+clrs+algorithmic+reasoning+benchmark%2E+In+International+Conference+on+Machine+Lear)