---
title: "WebArena: A Realistic Web Environment for — Zhou 2023"
description: "Most LLM-driven web agents are built and tested in simplified or frozen settings, such as toy sites or cached page snapshots."
authors: ["Shuyan Zhou", "Frank F. Xu", "Hao Zhu", "Xuhui Zhou", "Robert Lo", "Abishek Sridhar", "Xianyi Cheng", "Tianyue Ou", "Yonatan Bisk", "Daniel Fried", "Uri Alon", "Graham Neubig"]
year: 2023
venue: "arXiv:2307.13854"
source: "https://arxiv.org/abs/2307.13854"
doi: "10.48550/arxiv.2307.13854"
arxiv: "2307.13854"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / WebArena: A Realistic Web Environment for Building Autonomous Agents

# WebArena: A Realistic Web Environment for Building Autonomous Agents

**Authors:** [Shuyan Zhou](https://md.research.elamur.ai/free/en/authors/shuyan-zhou/), [Frank F. Xu](https://md.research.elamur.ai/free/en/authors/frank-f-xu/), [Hao Zhu](https://md.research.elamur.ai/free/en/authors/hao-zhu/), [Xuhui Zhou](https://md.research.elamur.ai/free/en/authors/xuhui-zhou/), [Robert Lo](https://md.research.elamur.ai/free/en/authors/robert-lo/), Abishek Sridhar, Xianyi Cheng, Tianyue Ou, [Yonatan Bisk](https://md.research.elamur.ai/free/en/authors/yonatan-bisk/), [Daniel Fried](https://md.research.elamur.ai/free/en/authors/daniel-fried/), Uri Alon, [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2023 · **Venue:** arXiv:2307.13854 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.13854) · [PDF](https://arxiv.org/pdf/2307.13854) · [DOI 10.48550/arxiv.2307.13854](https://doi.org/10.48550/arxiv.2307.13854) · [arXiv 2307.13854](https://arxiv.org/abs/2307.13854) · [OpenAlex](https://openalex.org/W4385327683)

**Citations:** 31 citations (openalex-snapshot · 2026-10-14) · [340](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most LLM-driven web agents are built and tested in simplified or frozen settings, such as toy sites or cached page snapshots. They are usually scored by matching their action strings against one reference path. WebArena replaces this with a self-hosted sandbox of working websites that can be reset to a fixed state. The sites are filled with data taken from real platforms: an online shop, a Reddit-style forum, a GitLab instance and a store-admin CMS. It also includes helper sites (a map, a calculator and a scratchpad) and reference material such as English Wikipedia and user manuals. Everything ships as Docker containers behind a gym-style interface. On top of this, the authors built a benchmark of 812 long-horizon tasks, each stated as a high-level natural-language goal. Programmatic checkers grade each task by inspecting the final site state or the returned answer, so any valid route to the goal counts. Few-shot LLM agents struggle: the best GPT-4 agent completes 14.41% of tasks, while humans reach 78.24%. This gives a reproducible way to measure progress toward practical web agents.

## Key points
- **Environment:** four self-hosted web apps (e-commerce "OneStopShop", Reddit-like forum, GitLab, shop-admin CMS), plus OpenStreetMap-based maps, a calculator, a scratchpad, and knowledge sources (Wikipedia, manuals). It runs in Docker with reset scripts and is available at webarena.dev.
- **Benchmark:** 812 long-horizon tasks with abstract, human-style intents. Scoring checks whether the goal was actually achieved, not whether the action sequence matches a reference, so different valid routes all count.
- **Agent interface:** the agent sees the URL, the open tabs and the page content (as a screenshot, the HTML DOM or the accessibility tree). WebArena is the first web environment to support tasks that span several tabs. Actions include click, type, hover, scroll, tab handling, goto/back, and stop with an answer.
- **Results:** few-shot agents built on GPT-4 and PaLM-2 were tested, including variants that reason before acting. The best one succeeded on 14.41% of tasks, against 78.24% for humans.
- **Failure analysis:** GPT-4 agents rely on the first relevant information they see and ignore fine detail, such as text already typed into a search box. They also explore little and rarely recover from mistakes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

340 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 319 not yet.

- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://md.research.elamur.ai/free/en/papers/chen-2023-agentverse-facilitating-multi-agent/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/)
- [ChatDev: Communicative Agents for Software Development](https://md.research.elamur.ai/free/en/papers/qian-2023-chatdev-communicative-agents-software/)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2024-medagents-large-language-models-as/)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://md.research.elamur.ai/free/en/papers/wang-2023-mint-evaluating-llms-multi-turn/)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://md.research.elamur.ai/free/en/papers/zeng-2023-agenttuning-enabling-generalized-agent/)
- [大语言模型的工具使用综述](https://md.research.elamur.ai/free/en/papers/zheng-2025-tool-use-large/)
- [Large Language Models for Information Retrieval: A Survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2023webarena,
  title = {{WebArena: A Realistic Web Environment for Building Autonomous Agents}},
  author = {Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham},
  year = {2023},
  doi = {10.48550/arxiv.2307.13854},
  eprint = {2307.13854},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13854}
}
```

- [← previous Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/)
- [next → What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ravi-2023-what-happens-before-after-multi/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

59 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Re… [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Qi+Wu%2C+Damien+Teney%2C+Jake+Bruce%2C+Mark+Johnson%2C+Niko+S%C3%BCnderhauf%2C+Ian+D%2E+Reid%2C+Stephen+Gould%2C+and+Anton+van+den+Hengel%2E+Vision%2Dand%2Dlanguage+navigation%3A+Interpreting+visually%2Dgrounded+nav) [our card](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/)
2. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00387. URL [http://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.html](http://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.html). — [doi:10.1109/cvpr.2018.00387](https://doi.org/10.1109/cvpr.2018.00387) [↗](http://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_Vision-and-Language_Navigation_Interpreting_CVPR_2018_paper.html)
3. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M%2E+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+Eric+Chu%2C+Jonathan+H%2E+Clark%2C+Laurent+El+Shafey%2C+Y) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
4. Yonatan Bisk, Jan Buys, Karl Pichotta, and Yejin Choi. Benchmarking hierarchical script knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4077–4085, Minne… [find](https://scholar.google.com/scholar?q=Yonatan+Bisk%2C+Jan+Buys%2C+Karl+Pichotta%2C+and+Yejin+Choi%2E+Benchmarking+hierarchical+script+knowledge%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio)
5. S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 82–9… [find](https://scholar.google.com/scholar?q=S%2ER%2EK%2E+Branavan%2C+Harr+Chen%2C+Luke+Zettlemoyer%2C+and+Regina+Barzilay%2E+Reinforcement+learning+for+mapping+instructions+to+actions%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+)
6. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. URL https://arxiv.org/abs/1606.01540. [↗](https://arxiv.org/abs/1606.01540)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. [↗](https://arxiv.org/abs/2107.03374) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. [find](https://scholar.google.com/scholar?q=Xiang+Deng%2C+Yu+Gu%2C+Boyuan+Zheng%2C+Shijie+Chen%2C+Samuel+Stevens%2C+Boshi+Wang%2C+Huan+Sun%2C+and+Yu+Su%2E+Mind2web%3A+Towards+a+generalist+agent+for+the+web%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/)
9. Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benc… [find](https://scholar.google.com/scholar?q=Linxi+Fan%2C+Guanzhi+Wang%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Yuncong+Yang%2C+Haoyi+Zhu%2C+Andrew+Tang%2C+De%2DAn+Huang%2C+Yuke+Zhu%2C+and+Anima+Anandkumar%2E+Minedojo%3A+Building+open%2Dended+embodied+agents+with+internet%2Dsc)
10. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764–10799. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+10764%E2%80%93)
11. Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: visual question answering in interactive environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4089–4098. [find](https://scholar.google.com/scholar?q=Daniel+Gordon%2C+Aniruddha+Kembhavi%2C+Mohammad+Rastegari%2C+Joseph+Redmon%2C+Dieter+Fox%2C+and+Ali+Farhadi%2E+IQA%3A+visual+question+answering+in+interactive+environments%2E+In+2018+IEEE+Conference+on+Computer+Visio)
12. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00430. URL [http://openaccess.thecvf.com/content_cvpr_2018/html/Gordon_IQA_Visual_Question_CVPR_2018_paper.html](http://openaccess.thecvf.com/content_cvpr_2018/html/Gordon_IQA_Visual_Question_CVPR_2018_paper.html). — [doi:10.1109/cvpr.2018.00430](https://doi.org/10.1109/cvpr.2018.00430) [↗](http://openaccess.thecvf.com/content_cvpr_2018/html/Gordon_IQA_Visual_Question_CVPR_2018_paper.html)
13. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023. [find](https://scholar.google.com/scholar?q=Izzeddin+Gur%2C+Hiroki+Furuta%2C+Austin+Huang%2C+Mustafa+Safdari%2C+Yutaka+Matsuo%2C+Douglas+Eck%2C+and+Aleksandra+Faust%2E+A+real%2Dworld+webagent+with+planning%2C+long+context+understanding%2C+and+program+synthesis%2E+ar)
14. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learn… [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Pieter+Abbeel%2C+Deepak+Pathak%2C+and+Igor+Mordatch%2E+Language+models+as+zero%2Dshot+planners%3A+Extracting+actionable+knowledge+for+embodied+agents%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+) [our card](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/)
15. PMLR, 2022. URL [https://proceedings.mlr.press/v162/huang22a.html](https://proceedings.mlr.press/v162/huang22a.html). [↗](https://proceedings.mlr.press/v162/huang22a.html)
16. Yacine Jernite, Kavya Srinet, Jonathan Gray, and Arthur Szlam. CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant. ArXiv preprint, abs/1905.01978, 2019. URL https://arxiv.org/abs/1905.01978. — [arXiv:1905.01978](https://arxiv.org/abs/1905.01978) [↗](https://arxiv.org/abs/1905.01978)
17. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. ArXiv preprint, abs/2303.17491, 2023. URL https://arxiv.org/abs/2303.17491. [↗](https://arxiv.org/abs/2303.17491) [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/)
18. Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017. [find](https://scholar.google.com/scholar?q=Eric+Kolve%2C+Roozbeh+Mottaghi%2C+Winson+Han%2C+Eli+VanderBilt%2C+Luca+Weihs%2C+Alvaro+Herrasti%2C+Daniel+Gordon%2C+Yuke+Zhu%2C+Abhinav+Gupta%2C+and+Ali+Farhadi%2E+AI2%2DTHOR%3A+An+Interactive+3D+Environment+for+Visual+AI%2E+a)
19. Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Informati… [find](https://scholar.google.com/scholar?q=Heinrich+K%C3%BCttler%2C+Nantas+Nardelli%2C+Alexander+H%2E+Miller%2C+Roberta+Raileanu%2C+Marco+Selvatici%2C+Edward+Grefenstette%2C+and+Tim+Rockt%C3%A4schel%2E+The+nethack+learning+environment%2E+In+Hugo+Larochelle%2C+Marc%E2%80%99Aurelio+)
20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+Matt) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
21. Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2E+A+path+towards+autonomous+machine+intelligence+version+0%2E9%2E+2%2C+2022%2D06%2D27%2E+Open+Review%2C+62%2C+2022%2E)
22. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp… [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Mandar+Joshi%2C+Iulia+Raluca+Turc%2C+Hexiang+Hu%2C+Fangyu+Liu%2C+Julian+Martin+Eisenschlos%2C+Urvashi+Khandelwal%2C+Peter+Shaw%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+Pix2struct%3A+Screenshot+parsing+a)
23. Xinze Li, Yixin Cao, Muhao Chen, and Aixin Sun. Take a break in the middle: Investigating subgoals towards hierarchical script generation. ArXiv preprint, abs/2305.10907, 2023. URL https://arxiv.org/abs/2305.10907. [↗](https://arxiv.org/abs/2305.10907)
24. Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198–8210, Online, 2020. Association for Computational Linguistics. doi… [find](https://scholar.google.com/scholar?q=Yang+Li%2C+Jiacong+He%2C+Xin+Zhou%2C+Yuan+Zhang%2C+and+Jason+Baldridge%2E+Mapping+natural+language+instructions+to+mobile+UI+action+sequences%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Co)
25. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493–9500. [find](https://scholar.google.com/scholar?q=Jacky+Liang%2C+Wenlong+Huang%2C+Fei+Xia%2C+Peng+Xu%2C+Karol+Hausman%2C+Brian+Ichter%2C+Pete+Florence%2C+and+Andy+Zeng%2E+Code+as+policies%3A+Language+model+programs+for+embodied+control%2E+In+2023+IEEE+International+Conf)
26. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings… [find](https://scholar.google.com/scholar?q=Evan+Zheran+Liu%2C+Kelvin+Guu%2C+Panupong+Pasupat%2C+Tianlin+Shi%2C+and+Percy+Liang%2E+Reinforcement+learning+on+web+interfaces+using+workflow%2Dguided+exploration%2E+In+6th+International+Conference+on+Learning+Rep)
27. OpenReview.net, 2018. URL [https://openreview.net/forum?id=ryTp3f-0-](https://openreview.net/forum?id=ryTp3f-0-). [↗](https://openreview.net/forum?id=ryTp3f-0-)
28. Jieyi Long. Large language model guided tree-of-thought. ArXiv preprint, abs/2305.08291, 2023. URL https://arxiv.org/abs/2305.08291. [↗](https://arxiv.org/abs/2305.08291)
29. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384–1403, Abu Dhabi, United Arab Emirates, 2022. Association for Computationa… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Yiming+Yang%2C+and+Graham+Neubig%2E+Language+models+of+code+are+few%2Dshot+commonsense+learners%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Lan)
30. Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281–300, 2016. [find](https://scholar.google.com/scholar?q=Dipendra+K+Misra%2C+Jaeyong+Sung%2C+Kevin+Lee%2C+and+Ashutosh+Saxena%2E+Tell+me+dave%3A+Context%2Dsensitive+grounding+of+natural+language+to+manipulation+instructions%2E+The+International+Journal+of+Robotics+Resear)
31. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. [find](https://scholar.google.com/scholar?q=Reiichiro+Nakano%2C+Jacob+Hilton%2C+Suchir+Balaji%2C+Jeff+Wu%2C+Long+Ouyang%2C+Christina+Kim%2C+Christopher+Hesse%2C+Shantanu+Jain%2C+Vineet+Kosaraju%2C+William+Saunders%2C+et+al%2E+Webgpt%3A+Browser%2Dassisted+question%2Danswer) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
32. OpenAI. Chatgpt: Optimizing language models for dialogue. 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Chatgpt%3A+Optimizing+language+models+for+dialogue%2E+2022%2E)
33. OpenAI. Gpt-4 technical report. arXiv, pp. 2303–08774, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2E+arXiv%2C+pp%2E+2303%E2%80%9308774%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
34. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
35. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8494–8502. [find](https://scholar.google.com/scholar?q=Xavier+Puig%2C+Kevin+Ra%2C+Marko+Boben%2C+Jiaman+Li%2C+Tingwu+Wang%2C+Sanja+Fidler%2C+and+Antonio+Torralba%2E+Virtualhome%3A+Simulating+household+activities+via+programs%2E+In+2018+IEEE+Conference+on+Computer+Vision+an)
36. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00886. URL [http://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html](http://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html). — [doi:10.1109/cvpr.2018.00886](https://doi.org/10.1109/cvpr.2018.00886) [↗](http://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html)
37. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, 2016. Association for Computational Linguistics. do… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+La) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
38. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, Melbourne, Australia, 2018. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
39. Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306.00245, 2023. — [arXiv:2306.00245](https://arxiv.org/abs/2306.00245) [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Mandar+Joshi%2C+James+Cohan%2C+Jonathan+Berant%2C+Panupong+Pasupat%2C+Hexiang+Hu%2C+Urvashi+Khandelwal%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+From+pixels+to+ui+actions%3A+Learning+to+follow+instructions)
40. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 201… [find](https://scholar.google.com/scholar?q=Tianlin+Shi%2C+Andrej+Karpathy%2C+Linxi+Fan%2C+Jonathan+Hernandez%2C+and+Percy+Liang%2E+World+of+bits%3A+An+open%2Ddomain+platform+for+web%2Dbased+agents%2E+In+Doina+Precup+and+Yee+Whye+Teh+%28eds%2E%29%2C+Proceedings+of+the+3)
41. PMLR, 2017. URL [http://proceedings.mlr.press/v70/shi17a.html](http://proceedings.mlr.press/v70/shi17a.html). [↗](http://proceedings.mlr.press/v70/shi17a.html)
42. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv preprint, abs/2303.11366, 2023. URL https://arxiv.org/abs/2303.11366. [↗](https://arxiv.org/abs/2303.11366) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
43. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA,… [find](https://scholar.google.com/scholar?q=Mohit+Shridhar%2C+Jesse+Thomason%2C+Daniel+Gordon%2C+Yonatan+Bisk%2C+Winson+Han%2C+Roozbeh+Mottaghi%2C+Luke+Zettlemoyer%2C+and+Dieter+Fox%2E+ALFRED%3A+A+benchmark+for+interpreting+grounded+instructions+for+everyday+tas) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/)
44. IEEE, 2020. doi: 10.1109/CVPR42600.2020.01075. URL https://doi.org/10.1109/CVPR42600.2020.01075. — [doi:10.1109/cvpr42600.2020.01075](https://doi.org/10.1109/cvpr42600.2020.01075) [↗](https://doi.org/10.1109/CVPR42600.2020.01075)
45. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. [find](https://scholar.google.com/scholar?q=Mohit+Shridhar%2C+Xingdi+Yuan%2C+Marc%2DAlexandre+C%C3%B4t%C3%A9%2C+Yonatan+Bisk%2C+Adam+Trischler%2C+and+Matthew+J%2E+Hausknecht%2E+Alfworld%3A+Aligning+text+and+embodied+environments+for+interactive+learning%2E+In+9th+Internatio) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/)
46. OpenReview.net, 2021. URL [https://openreview.net/forum?id=0IOX0YcCdTn](https://openreview.net/forum?id=0IOX0YcCdTn). [↗](https://openreview.net/forum?id=0IOX0YcCdTn)
47. Abishek Sridhar, Robert Lo, Frank F Xu, Hao Zhu, and Shuyan Zhou. Hierarchical prompting assists large language model on web navigation. arXiv preprint arXiv:2305.14257, 2023. [find](https://scholar.google.com/scholar?q=Abishek+Sridhar%2C+Robert+Lo%2C+Frank+F+Xu%2C+Hao+Zhu%2C+and+Shuyan+Zhou%2E+Hierarchical+prompting+assists+large+language+model+on+web+navigation%2E+arXiv+preprint+arXiv%3A2305%2E14257%2C+2023%2E)
48. Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. ArXiv preprint, abs/2105.13231, 2021. URL https://arxiv.org/abs/2105.13231. — [arXiv:2105.13231](https://arxiv.org/abs/2105.13231) [↗](https://arxiv.org/abs/2105.13231)
49. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv preprint, abs/2305.16291, 2023. URL https://arxiv.org/abs/2305.16291. [↗](https://arxiv.org/abs/2305.16291)
50. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. ArXiv preprint, abs/2212.10481, 2022. URL https://arxiv.org/abs/2212.10481. [↗](https://arxiv.org/abs/2212.10481)
51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+V+Le%2C+Denny+Zhou%2C+et+al%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+Advances+in+Neural+Informa) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
52. Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… [find](https://scholar.google.com/scholar?q=Nancy+Xu%2C+Sam+Masling%2C+Michael+Du%2C+Giovanni+Campagna%2C+Larry+Heck%2C+James+Landay%2C+and+Monica+Lam%2E+Grounding+open%2Ddomain+instructions+to+automate+web+support+tasks%2E+In+Proceedings+of+the+2021+Conference+)
53. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, B… [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+In+Proc) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
54. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. volume abs/2207.01206, 2022a. URL https://arxiv.org/abs/2207.01206. [↗](https://arxiv.org/abs/2207.01206) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/)
55. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022b. URL https://arxiv.org/abs/2210.03629. [↗](https://arxiv.org/abs/2210.03629) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
56. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, abs/2305.10601, 2023. URL https://arxiv.org/abs/2305.10601. [↗](https://arxiv.org/abs/2305.10601)
57. Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arxiv 2017. ArXiv preprint, abs/1709.00103, 2017. URL https://arxiv.org/abs/1709.00103. [↗](https://arxiv.org/abs/1709.00103)
58. Shuyan Zhou, Pengcheng Yin, and Graham Neubig. Hierarchical control of situated agents through natural language. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pp. 67–84, Seattle, USA, 2022a. Association for Computational Linguistics. doi: 10.18653/v1/202… [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Pengcheng+Yin%2C+and+Graham+Neubig%2E+Hierarchical+control+of+situated+agents+through+natural+language%2E+In+Proceedings+of+the+Workshop+on+Structured+and+Unstructured+Knowledge+Integration+%28SU)
59. Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, and Graham Neubig. Show me more details: Discovering hierarchies of procedures from semi-structured web data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Li+Zhang%2C+Yue+Yang%2C+Qing+Lyu%2C+Pengcheng+Yin%2C+Chris+Callison%2DBurch%2C+and+Graham+Neubig%2E+Show+me+more+details%3A+Discovering+hierarchies+of+procedures+from+semi%2Dstructured+web+data%2E+In+Proceed)