---
title: "DISCOVERYWORLD — paper by Jansen et al. (2024)"
description: "Testing whether AI agents can carry out science from start to finish is hard, because real laboratory… 36 references, cited by 36 papers here."
authors: ["Peter Jansen", "Marc-Alexandre Côté", "Tushar Khot", "Erin Bransom", "Bhavana Dalvi Mishra", "Bodhisattwa Prasad Majumder", "Oyvind Tafjord", "Peter Clark"]
year: 2024
venue: "arXiv:2406.06769"
source: "https://arxiv.org/abs/2406.06769"
doi: "10.48550/arxiv.2406.06769"
arxiv: "2406.06769"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jansen-2024-discoveryworld-virtual-environment-developing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jansen-2024-discoveryworld-virtual-environment-developing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

# DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

**Authors:** [Peter Jansen](https://md.research.elamur.ai/free/en/authors/peter-jansen/), [Marc-Alexandre Côté](https://md.research.elamur.ai/free/en/authors/marc-alexandre-cote/), [Tushar Khot](https://md.research.elamur.ai/free/en/authors/tushar-khot/), [Erin Bransom](https://md.research.elamur.ai/free/en/authors/erin-bransom/), [Bhavana Dalvi Mishra](https://md.research.elamur.ai/free/en/authors/bhavana-dalvi-mishra/), [Bodhisattwa Prasad Majumder](https://md.research.elamur.ai/free/en/authors/bodhisattwa-prasad-majumder/), [Oyvind Tafjord](https://md.research.elamur.ai/free/en/authors/oyvind-tafjord/) and [Peter Clark](https://md.research.elamur.ai/free/en/authors/peter-clark/) · **Year:** 2024 · **Venue:** [arXiv:2406.06769](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.06769) · [PDF](https://arxiv.org/pdf/2406.06769) · [DOI 10.48550/arxiv.2406.06769](https://doi.org/10.48550/arxiv.2406.06769) · [arXiv 2406.06769](https://arxiv.org/abs/2406.06769) · [OpenAlex](https://openalex.org/W4415798813)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/jansen-2024-discoveryworld-virtual-environment-developing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Testing whether AI agents can carry out science from start to finish is hard, because real laboratory experiments are costly and push systems toward narrow, task-specific solutions. The authors address this with DISCOVERYWORLD, a cheap simulated environment. It is text-based, with an optional 2D top-down view. Agents move through a world, handle objects, use instruments and record observations. To succeed, they must come up with hypotheses, design and run experiments, interpret the data and then act on what they learned. The benchmark has 120 tasks drawn from eight themes, ranging from proteomics to rocket science. Scoring is automatic and covers three aspects: whether the task was finished, whether relevant actions were taken, and whether the agent arrived at the correct explanation. Three LLM-based baselines that do well on earlier environments performed poorly here. Human scientists, by contrast, found the tasks difficult but mostly solvable. This gap makes the resource a useful testbed for building general-purpose discovery agents.

## Key points
- **Task design:** There are 8 themes (Proteomics, Chemistry, Archaeology/radioisotope dating, Reactor Lab, Plant Nutrients, Space Sick, Rocket Science, Translation). Each comes in 3 difficulty levels with parametric variants, giving 120 tasks, plus simpler Unit Test tasks for individual skills.
- **Baseline agents:** The three agents were ReAct, Plan+Exec (iterative decomposition) and Hypothesizer, which keeps a CLIN-style memory of hypotheses and measurements.
- **Agent results:** ReAct had the highest completion rates, at 38% on easy and 18% on challenge tasks. Hypothesizer was best at recovering explanatory knowledge, but reached only 34% on easy and 8% on challenge tasks. On Unit Tests, agents completed around 60% or more, which suggests they can handle component skills but not full discovery cycles.
- **Human comparison:** 11 scientists with an MSc or PhD, recruited through UpWork, averaged 66% task completion and 55% knowledge score.
- **Availability and context:** The code is released under Apache-2.0 at github.com/allenai/discoveryworld. The authors present it as harder than ScienceWorld, because tasks demand open-ended ideation and systematic search.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 96 works

## Cite

Cite as: Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord and Peter Clark. 2024. DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents. arXiv:2406.06769.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jansen2024discoveryworld,
  title = {{DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents}},
  author = {Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord and Peter Clark},
  year = {2024},
  doi = {10.48550/arxiv.2406.06769},
  eprint = {2406.06769},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.06769}
}
```

- [← previous Polysemy–-Evidence from Linguistics, Behavioral Science, and Contextualized Language Models](https://md.research.elamur.ai/free/en/papers/haber-2024-polysemy-evidence-linguistics-behavioral-science/)
- [next → Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning](https://md.research.elamur.ai/free/en/papers/bi-2024-forest-thought-scaling-test-time/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

36 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. [Autonomous chemical research with large language models](https://md.research.elamur.ai/free/en/papers/boiko-2023-autonomous-chemical-research-large-language/). Nature, 624:570 – 578, 2023. URL [https://api.semanticscholar.org/CorpusID:266432059](https://api.semanticscholar.org/CorpusID:266432059). [↗](https://api.semanticscholar.org/CorpusID:266432059)
2. A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6:525 – 535, 2023. URL [https://api.semanticscholar.org/CorpusID:258059792](https://api.semanticscholar.org/CorpusID:258059792). [↗](https://api.semanticscholar.org/CorpusID:258059792)
3. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016. [find](https://scholar.google.com/scholar?q=G%2E+Brockman%2C+V%2E+Cheung%2C+L%2E+Pettersson%2C+J%2E+Schneider%2C+J%2E+Schulman%2C+J%2E+Tang%2C+and+W%2E+Zaremba%2E+Openai+gym%2C+2016%2E)
4. M. Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. [Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks](https://md.research.elamur.ai/free/en/papers/chevalier-boisvert-2023-minigrid-miniworld-modular-customizable-reinforcement/). CoRR, abs/2306.13831, 2023.
5. M.-A. Côté, Á. Kádár, X. Yuan, B. A. Kybartas, T. Barnes, E. Fine, J. Moore, M. J. Hausknecht, L. E. Asri, M. Adada, W. Tay, and A. Trischler. [TextWorld: A learning environment for text-based games](https://md.research.elamur.ai/free/en/papers/cote-2018-textworld-learning-environment-text-based/). In CGW@IJCAI, 2018. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:49552345. [↗](https://api.semanticscholar.org/)
6. L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. [MineDojo: Building open-ended embodied agents with internet-scale knowledge](https://md.research.elamur.ai/free/en/papers/fan-2022-minedojo-building-open-ended-embodied/). In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https:/…
7. M. J. Hausknecht, P. Ammanabrolu, M.-A. Côté, and X. Yuan. Interactive fiction games: A colossal adventure. In AAAI Conference on Artificial Intelligence, 2019. URL [https://api.semanticscholar.org/CorpusID:202565447](https://api.semanticscholar.org/CorpusID:202565447). [↗](https://api.semanticscholar.org/CorpusID:202565447)
8. Q. Huang, J. Vora, P. Liang, and J. Leskovec. [Mlagentbench: Evaluating language agents on machine learning experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). arXiv preprint arXiv:2310.03302, 2024. URL [https://api.semanticscholar.org/CorpusID:263671541](https://api.semanticscholar.org/CorpusID:263671541). [↗](https://api.semanticscholar.org/CorpusID:263671541)
9. Infocom. Learning zil, 1989. URL [https://web.archive.org/web/20100807003406/](https://web.archive.org/web/20100807003406/) http://xlisp.org/zil.pdf. Accessed: 2024-06-05. [↗](https://web.archive.org/web/20100807003406/)
10. J. M. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M.… — [Highly accurate protein structure prediction with AlphaFold](https://md.research.elamur.ai/free/en/papers/jumper-2021-highly-accurate-protein-structure-prediction/)
11. T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal. [Decomposed prompting: A modular approach for solving complex tasks](https://md.research.elamur.ai/free/en/papers/khot-2022-decomposed-prompting-modular-approach/). In ICLR, 2023.
12. R. D. King, K. E. Whelan, F. M. Jones, P. G. K. Reiser, C. H. Bryant, S. H. Muggleton, D. B. Kell, and S. G. Oliver. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427:247–252, 2004. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:4428725. [↗](https://api.semanticscholar.org/)
13. H. Kitano. Artificial intelligence to win the nobel prize and beyond: Creating the engine for scientific discovery. AI magazine, 37(1):39–49, 2016. [find](https://scholar.google.com/scholar?q=H%2E+Kitano%2E+Artificial+intelligence+to+win+the+nobel+prize+and+beyond%3A+Creating+the+engine+for+scientific+discovery%2E+AI+magazine%2C+37%281%29%3A39%E2%80%9349%2C+2016%2E)
14. E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. K. Gupta, and A. Farhadi. [AI2-THOR: An interactive 3d environment for visual ai](https://md.research.elamur.ai/free/en/papers/kolve-2017-ai2-thor-interactive-3d-environment/). ArXiv, abs/1712.05474, 2017. URL [https://api.semanticscholar.org/CorpusID:28328610](https://api.semanticscholar.org/CorpusID:28328610). [↗](https://api.semanticscholar.org/CorpusID:28328610)
15. R. Krishna, J. Wang, W. Ahern, P. Sturmfels, P. Venkatesh, I. Kalvet, G. R. Lee, F. S. Morey-Burrows, I. V. Anishchenko, I. R. Humphreys, R. McHugh, D. Vafeados, X. Li, G. A. Sutherland, A. Hitchcock, C. N. Hunter, A. Kang, E. Brackenbrough, A. K. Bera, M. Baek, F. DiMaio, and D. Baker. Generalized… [find](https://scholar.google.com/scholar?q=R%2E+Krishna%2C+J%2E+Wang%2C+W%2E+Ahern%2C+P%2E+Sturmfels%2C+P%2E+Venkatesh%2C+I%2E+Kalvet%2C+G%2E+R%2E+Lee%2C+F%2E+S%2E+Morey%2DBurrows%2C+I%2E+V%2E+Anishchenko%2C+I%2E+R%2E+Humphreys%2C+R%2E+McHugh%2C+D%2E+Vafeados%2C+X%2E+Li%2C+G%2E+A%2E+Sutherland%2C+A%2E+Hitchcock%2C)
16. H. Küttler, N. Nardelli, A. Miller, R. Raileanu, M. Selvatici, E. Grefenstette, and T. Rocktäschel. The NetHack learning environment. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7671– 7684. Curran Assoc… [find](https://scholar.google.com/scholar?q=H%2E+K%C3%BCttler%2C+N%2E+Nardelli%2C+A%2E+Miller%2C+R%2E+Raileanu%2C+M%2E+Selvatici%2C+E%2E+Grefenstette%2C+and+T%2E+Rockt%C3%A4schel%2E+The+NetHack+learning+environment%2E+In+H%2E+Larochelle%2C+M%2E+Ranzato%2C+R%2E+Hadsell%2C+M%2E+Balcan%2C+and+H%2E+Lin%2C+e)
17. P. Langley. Scientific discovery: Computational explorations of the creative processes. MIT press, 1987. [find](https://scholar.google.com/scholar?q=P%2E+Langley%2E+Scientific+discovery%3A+Computational+explorations+of+the+creative+processes%2E+MIT+press%2C+1987%2E)
18. B. P. Majumder, B. Dalvi, P. A. Jansen, O. Tafjord, N. Tandon, L. Zhang, C. Callison-Burch, and P. Clark. [CLIN: A continually learning language agent for rapid task adaptation and generalization](https://md.research.elamur.ai/free/en/papers/majumder-2023-clin-continually-learning-language-agent/). ArXiv, abs/2310.10134, 2023. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:264146262. [↗](https://api.semanticscholar.org/)
19. K. Popper. The logic of scientific discovery. Routledge, 2005. [find](https://scholar.google.com/scholar?q=K%2E+Popper%2E+The+logic+of+scientific+discovery%2E+Routledge%2C+2005%2E)
20. A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot. [ADaPT: As-needed decomposition and planning with language models](https://md.research.elamur.ai/free/en/papers/prasad-2024-adapt-as-needed-decomposition-planning/). In NAACL (Findings), 2023.
21. Pygame. Pygame. [https://www.pygame.org/](https://www.pygame.org/), 2024. Accessed: 2024-06-05. [↗](https://www.pygame.org/)
22. B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, A. Fawzi, J. Grochow, A. Lodi, J.-B. Mouret, T. Ringer, and T. Yu. [Mathematical discoveries from program search with large language models](https://md.research.elamur.ai/free/en/papers/paredes-2023-mathematical-discoveries-program-search-large/). Nature, 625:468 – 4…
23. M. Samvelyan, R. Kirk, V. Kurin, J. Parker-Holder, M. Jiang, E. Hambro, F. Petroni, H. Küttler, E. Grefenstette, and T. Rocktäschel. MiniHack the planet: A sandbox for open-ended reinforcement learning research. ArXiv, abs/2109.13202, 2021. URL [https://api.semanticscholar.org/CorpusID:237274213](https://api.semanticscholar.org/CorpusID:237274213). — [arXiv:2109.13202](https://arxiv.org/abs/2109.13202) [↗](https://api.semanticscholar.org/CorpusID:237274213)
24. A. D. Sendek, E. D. Cubuk, E. R. Antoniuk, G. Cheon, Y. Cui, and E. J. Reed. Machine learning-assisted discovery of solid li-ion conducting materials. Chemistry of Materials, 2018. URL [https://api.semanticscholar.org/CorpusID:139501995](https://api.semanticscholar.org/CorpusID:139501995). [↗](https://api.semanticscholar.org/CorpusID:139501995)
25. M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/). In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL https://arxiv.org/abs/1912.01734. [↗](https://arxiv.org/abs/1912.01734)
26. M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. J. Hausknecht. [ALFWorld: Aligning text and embodied environments for interactive learning](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/). ArXiv, abs/2010.03768, 2020. URL [https://api.semanticscholar.org/CorpusID:222208810](https://api.semanticscholar.org/CorpusID:222208810). [↗](https://api.semanticscholar.org/CorpusID:222208810)
27. M. Towers, J. K. Terry, A. Kwiatkowski, J. U. Balis, G. d. Cola, T. Deleu, M. Goulão, A. Kallinteris, A. KG, M. Krimmel, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. J. Tai, A. T. J. Shen, and O. G. Younis. Gymnasium, Mar. 2023. URL [https://zenodo.org/record/](https://zenodo.org/record/) 8127025. [↗](https://zenodo.org/record/)
28. J. Wang, M. King, N. Porcel, Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy, M. Cassin, M. Reynolds, F. Song, G. Buttimore, D. Reichert, N. Rabinowitz, L. Matthey, D. Hassabis, A. Lerchner, and M. Botvinick. Alchemy: A structured task distribution for meta-reinforcement learning. arXiv preprint arXiv:210… — [arXiv:2102.02926](https://arxiv.org/abs/2102.02926) [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+M%2E+King%2C+N%2E+Porcel%2C+Z%2E+Kurth%2DNelson%2C+T%2E+Zhu%2C+C%2E+Deck%2C+P%2E+Choy%2C+M%2E+Cassin%2C+M%2E+Reynolds%2C+F%2E+Song%2C+G%2E+Buttimore%2C+D%2E+Reichert%2C+N%2E+Rabinowitz%2C+L%2E+Matthey%2C+D%2E+Hassabis%2C+A%2E+Lerchner%2C+and+M%2E+Botvinic)
29. L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In ACL, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Wang%2C+W%2E+Xu%2C+Y%2E+Lan%2C+Z%2E+Hu%2C+Y%2E+Lan%2C+R%2E+K%2E%2DW%2E+Lee%2C+and+E%2E%2DP%2E+Lim%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+In+ACL%2C+2023%2E)
30. R. Wang, P. Jansen, M.-A. Côté, and P. Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Wang%2C+P%2E+Jansen%2C+M%2E%2DA%2E+C%C3%B4t%C3%A9%2C+and+P%2E+Ammanabrolu%2E+ScienceWorld%3A+Is+your+agent+smarter+than+a+5th+grader%3F+arXiv+preprint+arXiv%3A2203%2E07540%2C+2022%2E)
31. R. Wang, P. A. Jansen, M.-A. Côté, and P. Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Conference on Empirical Methods in Natural Language Processing, 2022. URL [https://api.semanticscholar.org/CorpusID:247451124](https://api.semanticscholar.org/CorpusID:247451124). [↗](https://api.semanticscholar.org/CorpusID:247451124)
32. K. A. Williams, E. Bilsland, A. Sparkes, W. Aubrey, M. Young, L. N. Soldatova, K. D. Grave, J. Ramon, M. de Clare, W. Sirawaraporn, S. G. Oliver, and R. D. King. Cheaper faster drug development validated by the repositioning of drugs against neglected tropical diseases. Journal of the Royal Society… [find](https://scholar.google.com/scholar?q=K%2E+A%2E+Williams%2C+E%2E+Bilsland%2C+A%2E+Sparkes%2C+W%2E+Aubrey%2C+M%2E+Young%2C+L%2E+N%2E+Soldatova%2C+K%2E+D%2E+Grave%2C+J%2E+Ramon%2C+M%2E+de+Clare%2C+W%2E+Sirawaraporn%2C+S%2E+G%2E+Oliver%2C+and+R%2E+D%2E+King%2E+Cheaper+faster+drug+development+valida)
33. M. Xu, G. Jiang, W. Liang, C. Zhang, and Y. Zhu. Interactive visual reasoning under uncertainty, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Xu%2C+G%2E+Jiang%2C+W%2E+Liang%2C+C%2E+Zhang%2C+and+Y%2E+Zhu%2E+Interactive+visual+reasoning+under+uncertainty%2C+2023%2E)
34. J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. [InterCode: Standardizing and benchmarking interactive coding with execution feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/). In Advances in Neural Information Processing Systems, volume 36, 2023.
35. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. [ReAct: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). ArXiv, abs/2210.03629, 2022. URL [https://api.semanticscholar.org/CorpusID:252762395](https://api.semanticscholar.org/CorpusID:252762395). [↗](https://api.semanticscholar.org/CorpusID:252762395)
36. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. [WebArena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/), 2024.

*Updated 3 November 2026*