---
title: "AndroidWorld — paper by Rawles et al. (2024)"
description: "The authors introduce AndroidWorld, an open environment built on the Android Emulator and the… 85 references, cited by 80 papers here."
authors: ["Christopher Rawles", "Sarah Clinckemaillie", "Yifan Chang", "Jonathan Waltz", "Gabrielle Lau", "Marybeth Fair", "Alice Li", "William Bishop", "Wei Li", "Folawiyo Campbell-Ajala", "Daniel Toyama", "Robert Berry", "Divya Tyamagundlu", "Timothy Lillicrap", "Oriana Riva"]
year: 2024
venue: "arXiv:2405.14573"
source: "https://arxiv.org/abs/2405.14573"
doi: "10.48550/arxiv.2405.14573"
arxiv: "2405.14573"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rawles-2024-androidworld-dynamic-benchmarking-environment-autonomous/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rawles-2024-androidworld-dynamic-benchmarking-environment-autonomous/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

# AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

**Authors:** [Christopher Rawles](https://md.research.elamur.ai/free/en/authors/chris-rawles/), Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, [Alice Li](https://md.research.elamur.ai/free/en/authors/alice-li/), [William Bishop](https://md.research.elamur.ai/free/en/authors/william-bishop/), [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/), [Folawiyo Campbell-Ajala](https://md.research.elamur.ai/free/en/authors/folawiyo-campbell-ajala/), Daniel Toyama, Robert Berry, Divya Tyamagundlu, [Timothy Lillicrap](https://md.research.elamur.ai/free/en/authors/timothy-lillicrap/) and Oriana Riva · **Year:** 2024 · **Venue:** arXiv:2405.14573 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.14573) · [PDF](https://arxiv.org/pdf/2405.14573) · [DOI 10.48550/arxiv.2405.14573](https://doi.org/10.48550/arxiv.2405.14573) · [arXiv 2405.14573](https://arxiv.org/abs/2405.14573) · [OpenAlex](https://openalex.org/W4398836283)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [80](https://md.research.elamur.ai/free/en/papers/rawles-2024-androidworld-dynamic-benchmarking-environment-autonomous/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Evaluating agents that control devices from natural-language instructions is difficult. Step-by-step comparison with a recorded human demonstration penalizes valid alternative routes. Existing environments with automatic rewards mostly target desktops or the web and use fixed test cases. The authors introduce AndroidWorld, an open environment built on the Android Emulator and the AndroidEnv library. Its tasks run in real Android apps and are generated from templates with randomly sampled parameters, so the number of distinct goals is practically unlimited. Success is judged by inspecting the device's actual state, such as app databases, the filesystem and system APIs. Every task has its own setup, success check and cleanup, which keeps runs reproducible. MiniWoB++ web tasks are also bundled in. The authors' multimodal baseline, M3A, solves 30.6% of tasks, far below the 80.0% human rate, and an adapted desktop web agent does worse on mobile. Varying task parameters causes large swings in success, which shows that a single fixed test set can overstate how reliable an agent is.

## Key points
- AndroidWorld contains 116 programmatic, parameterized task templates spanning 20 real-world Android apps. It also integrates 92 MiniWoB++ tasks.
- Rewards are derived from device state through database queries, filesystem checks, system API calls and string matching of answers. Tasks fall into two types: task completion (TC), where the agent must perform an action, and information retrieval (IR), where it must answer a question.
- The setup is lightweight, needing about 2 GB of RAM and 8 GB of disk. It is released at github.com/google-research/android_world.
- M3A (Multimodal Autonomous Agent for Android) sets the best reported result at 30.6% success versus 80.0% for humans. Adding visual input to M3A generally did not beat its text-only variant.
- Robustness tests show that agent performance varies considerably across task variations, mainly because of changes in intent parameters.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 160 works

## Cite

Cite as: Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap and Oriana Riva. 2024. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv:2405.14573.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rawles2024androidworld,
  title = {{AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents}},
  author = {Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap and Oriana Riva},
  year = {2024},
  doi = {10.48550/arxiv.2405.14573},
  eprint = {2405.14573},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.14573}
}
```

- [← previous Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://md.research.elamur.ai/free/en/papers/zhou-2024-transfusion-predict-next-token-diffuse/)
- [next → CodeGemma: Open Code Models Based on Gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

85 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Timothy Lillicrap, Alistair Muldal, Blake Richards, Adam Santoro, Tamara von Glehn, Greg Wayne, Nathaniel Wong, and Chen Yan. Evaluating multimodal interactive agents, 2022. [find](https://scholar.google.com/scholar?q=Josh+Abramson%2C+Arun+Ahuja%2C+Federico+Carnevale%2C+Petko+Georgiev%2C+Alex+Goldin%2C+Alden+Hung%2C+Jessica+Landon%2C+Timothy+Lillicrap%2C+Alistair+Muldal%2C+Blake+Richards%2C+Adam+Santoro%2C+Tamara+von+Glehn%2C+Greg+Wayne%2C+)
2. William E Bishop, Alice Li, Christopher Rawles, and Oriana Riva. Latent state estimation helps ui agents to reason, 2024. [find](https://scholar.google.com/scholar?q=William+E+Bishop%2C+Alice+Li%2C+Christopher+Rawles%2C+and+Oriana+Riva%2E+Latent+state+estimation+helps+ui+agents+to+reason%2C+2024%2E)
3. Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale, 2024. URL https://arxiv.org/abs/2409.08264. — [arXiv:2409.08264](https://arxiv.org/abs/2409.08264) [↗](https://arxiv.org/abs/2409.08264)
4. Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interactive visual environments. CoRR, abs/2104.08560, 2021. URL https://arxiv.org/abs/ 2104.08560. — [arXiv:2104.08560](https://arxiv.org/abs/2104.08560) [↗](https://arxiv.org/abs/)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. [Seeclick: Harnessing gui grounding for advanced visual gui agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). arXiv preprint arXiv:2401.10935, 2024.
7. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). arXiv preprint arXiv:2403.04132, 2024.
8. Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv:1806.08295, 2018. [find](https://scholar.google.com/scholar?q=Ce%CC%81dric+Colas%2C+Olivier+Sigaud%2C+and+Pierre%2DYves+Oudeyer%2E+How+many+random+seeds%3F+statistical+power+analysis+in+deep+reinforcement+learning+experiments%2E+arXiv+preprint+arXiv%3A1806%2E08295%2C+2018%2E)
9. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. [Mind2Web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/), 2023.
10. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024. — [arXiv:2403.07718](https://arxiv.org/abs/2403.07718) [find](https://scholar.google.com/scholar?q=Alexandre+Drouin%2C+Maxime+Gasse%2C+Massimo+Caccia%2C+Issam+H+Laradji%2C+Manuel+Del+Verme%2C+Tom+Marty%2C+Le%CC%81o+Boisvert%2C+Megh+Thakkar%2C+Quentin+Cappart%2C+David+Vazquez%2C+et+al%2E+Workarena%3A+How+capable+are+web+agents+)
11. Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-Language models as success detectors. March 2023. [find](https://scholar.google.com/scholar?q=Yuqing+Du%2C+Ksenia+Konyushkova%2C+Misha+Denil%2C+Akhil+Raju%2C+Jessica+Landon%2C+Felix+Hill%2C+Nando+de+Freitas%2C+and+Serkan+Cabi%2E+Vision%2DLanguage+models+as+success+detectors%2E+March+2023%2E)
12. Nicholas Farn and Richard Shin. Tooltalk: Evaluating tool-usage in a conversational setting. arXiv preprint arXiv:2311.10775, 2023. — [arXiv:2311.10775](https://arxiv.org/abs/2311.10775) [find](https://scholar.google.com/scholar?q=Nicholas+Farn+and+Richard+Shin%2E+Tooltalk%3A+Evaluating+tool%2Dusage+in+a+conversational+setting%2E+arXiv+preprint+arXiv%3A2311%2E10775%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E10775%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E10775%29)
13. Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with Instruction-Finetuned foundation models. May 2023. [find](https://scholar.google.com/scholar?q=Hiroki+Furuta%2C+Kuang%2DHuei+Lee%2C+Ofir+Nachum%2C+Yutaka+Matsuo%2C+Aleksandra+Faust%2C+Shixiang+Shane+Gu%2C+and+Izzeddin+Gur%2E+Multimodal+web+navigation+with+Instruction%2DFinetuned+foundation+models%2E+May+2023%2E)
14. Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. Assistgui: Task-oriented desktop graphical user interface automation. arXiv preprint arXiv:2312.13108, 2023. — [arXiv:2312.13108](https://arxiv.org/abs/2312.13108) [find](https://scholar.google.com/scholar?q=Difei+Gao%2C+Lei+Ji%2C+Zechen+Bai%2C+Mingyu+Ouyang%2C+Peiran+Li%2C+Dongxing+Mao%2C+Qinchen+Wu%2C+Weichen+Zhang%2C+Peiyi+Wang%2C+Xiangwu+Guo%2C+et+al%2E+Assistgui%3A+Task%2Doriented+desktop+graphical+user+interface+automation%2E+)
15. Gemini. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/), 2023.
16. Significant Gravitas. AutoGPT. [https://agpt.co](https://agpt.co), 2023. https://agpt.co. [↗](https://agpt.co)
17. Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning, 2022a. [find](https://scholar.google.com/scholar?q=Izzeddin+Gur%2C+Natasha+Jaques%2C+Yingjie+Miao%2C+Jongwook+Choi%2C+Manoj+Tiwari%2C+Honglak+Lee%2C+and+Aleksandra+Faust%2E+Environment+generation+for+zero%2Dshot+compositional+reinforcement+learning%2C+2022a%2E)
18. Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint arXiv:2210.03945, 2022b. [find](https://scholar.google.com/scholar?q=Izzeddin+Gur%2C+Ofir+Nachum%2C+Yingjie+Miao%2C+Mustafa+Safdari%2C+Austin+Huang%2C+Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Noah+Fiedel%2C+and+Aleksandra+Faust%2E+Understanding+html+with+large+language+models%2E+arXiv+prep)
19. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. [Webvoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). arXiv preprint arXiv:2401.13919, 2024.
20. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Riashat+Islam%2C+Philip+Bachman%2C+Joelle+Pineau%2C+Doina+Precup%2C+and+David+Meger%2E+Deep+reinforcement+learning+that+matters%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence)
21. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). arXiv preprint arXiv:2312.08914, 2023.
22. Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang… [find](https://scholar.google.com/scholar?q=Peter+C+Humphreys%2C+David+Raposo%2C+Tobias+Pohlen%2C+Gregory+Thornton%2C+Rachita+Chhaparia%2C+Alistair+Muldal%2C+Josh+Abramson%2C+Petko+Georgiev%2C+Adam+Santoro%2C+and+Timothy+Lillicrap%2E+A+data%2Ddriven+approach+for+lea)
23. Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553, 2024. [find](https://scholar.google.com/scholar?q=Raghav+Kapoor%2C+Yash+Parag+Butala%2C+Melisa+Russak%2C+Jing+Yu+Koh%2C+Kiran+Kamble%2C+Waseem+Alshikh%2C+and+Ruslan+Salakhutdinov%2E+Omniact%3A+A+dataset+and+benchmark+for+enabling+multimodal+generalist+autonomous+age)
24. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. [Language models can solve computer tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/). Advances in Neural Information Processing Systems, 36, 2024.
25. Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, et al. Evaluating language-model agents on realistic autonomous tasks. arXiv preprint arXiv:2312.11671, 2023. — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) [find](https://scholar.google.com/scholar?q=Megan+Kinniment%2C+Lucas+Jun+Koba+Sato%2C+Haoxing+Du%2C+Brian+Goodrich%2C+Max+Hasin%2C+Lawrence+Chan%2C+Luke+Harold+Miles%2C+Tao+R+Lin%2C+Hjalmar+Wijk%2C+Joel+Burget%2C+et+al%2E+Evaluating+language%2Dmodel+agents+on+realisti)
26. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. [Visualwebarena: Evaluating multimodal agents on realistic visual web tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/). arXiv preprint arXiv:2401.13649, 2024.
27. Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent. arXiv preprint arXiv:2404.[03648, 2024](https://md.research.elamur.ai/free/en/papers/lai-2024-autowebglm-large-language-model-based/).
28. Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations. In ICLR 2024 Workshop on Generative Models for Decision Making, 2024. [find](https://scholar.google.com/scholar?q=Juyong+Lee%2C+Taywon+Min%2C+Minyong+An%2C+Changyeon+Kim%2C+and+Kimin+Lee%2E+Benchmarking+mobile+device+control+agents+across+diverse+configurations%2E+In+ICLR+2024+Workshop+on+Generative+Models+for+Decision+Makin)
29. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. [API-Bank: A comprehensive benchmark for Tool-Augmented LLMs](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/). April 2023a.
30. Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A Zero-Shot language agent for computer control with structured reflection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11261–11274, Singapore, December 2023b. A… [find](https://scholar.google.com/scholar?q=Tao+Li%2C+Gang+Li%2C+Zhiwei+Deng%2C+Bryan+Wang%2C+and+Yang+Li%2E+A+Zero%2DShot+language+agent+for+computer+control+with+structured+reflection%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Findings+of+the+)
31. Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. In Advances in Neural Information Processing Systems (NeurIPS 2024), 2024. URL https://arxiv.org/abs/ 2406.03679. [↗](https://arxiv.org/abs/)
32. Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. [Mapping natural language instructions to mobile UI action sequences](https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/). In Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8198–8210. Association for Computational L…
33. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. [Reinforcement learning on web interfaces using workflow-guided exploration](https://md.research.elamur.ai/free/en/papers/liu-2018-reinforcement-learning-web-interfaces-using/). In 6th International Conference on Learning Representations (ICLR ’18), 2018a. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) ryTp3f-0-. [↗](https://openreview.net/forum?id=)
34. Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learning design semantics for mobile apps. In Proc. of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST ’18, pp. 569–579, New York, NY, USA, 2018b. Association for Computing Machinery.… — [doi:10.1145/3242587.3242650](https://doi.org/10.1145/3242587.3242650) [find](https://scholar.google.com/scholar?q=Thomas+F%2E+Liu%2C+Mark+Craft%2C+Jason+Situ%2C+Ersin+Yumer%2C+Radomir+Mech%2C+and+Ranjitha+Kumar%2E+Learning+design+semantics+for+mobile+apps%2E+In+Proc%2E+of+the+31st+Annual+ACM+Symposium+on+User+Interface+Software+an)
35. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. [G-Eval: NLG evaluation using GPT-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/), 2023.
36. Xing Han Lù, Zdeněk Kasner, and Siva Reddy. [WebLINX: Real-World website navigation with Multi-Turn dialogue](https://md.research.elamur.ai/free/en/papers/lu-2024-weblinx-real-world-website-navigation/). February 2024.
37. Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. [Eureka: Human-level reward design via coding large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-eureka-human-level-reward-design/), 2023.
38. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedbac)
39. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. [GAIA: a benchmark for general AI assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/). November 2023.
40. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglu%2C+David+Silver%2C+Alex+Graves%2C+Ioannis+Antonoglou%2C+Daan+Wierstra%2C+and+Martin+Riedmiller%2E+Playing+atari+with+deep+reinforcement+learning%2C+2013%2E)
41. Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. BAGEL: Bootstrapping agents by guiding exploration with language, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2403.08140. [↗](https://arxiv.org/)
42. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
43. Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. April 2024. [find](https://scholar.google.com/scholar?q=Jiayi+Pan%2C+Yichi+Zhang%2C+Nicholas+Tomlin%2C+Yifei+Zhou%2C+Sergey+Levine%2C+and+Alane+Suhr%2E+Autonomous+evaluation+and+refinement+of+digital+agents%2E+April+2024%2E)
44. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. [find](https://scholar.google.com/scholar?q=Antonin+Raffin%2C+Ashley+Hill%2C+Adam+Gleave%2C+Anssi+Kanervisto%2C+Maximilian+Ernestus%2C+and+Noah+Dormann%2E+Stable%2Dbaselines3%3A+Reliable+reinforcement+learning+implementations%2E+Journal+of+Machine+Learning+Resea)
45. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088, 2023. — [arXiv:2307.10088](https://arxiv.org/abs/2307.10088) [find](https://scholar.google.com/scholar?q=Christopher+Rawles%2C+Alice+Li%2C+Daniel+Rodriguez%2C+Oriana+Riva%2C+and+Timothy+Lillicrap%2E+Android+in+the+wild%3A+A+large%2Dscale+dataset+for+android+device+control%2E+arXiv+preprint+arXiv%3A2307%2E10088%2C+2023%2E+%E2%80%94+%5BarX)
46. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. [Identifying the risks of LM agents with an LM-Emulated sandbox](https://md.research.elamur.ai/free/en/papers/ruan-2023-identifying-risks-lm-agents-lm/). September 2023.
47. Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to UI actions: Learning to follow instructions via graphical user interfaces. May 2023. [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Mandar+Joshi%2C+James+Cohan%2C+Jonathan+Berant%2C+Panupong+Pasupat%2C+Hexiang+Hu%2C+Urvashi+Khandelwal%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+From+pixels+to+UI+actions%3A+Learning+to+follow+instructions)
48. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proc. of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, p… [find](https://scholar.google.com/scholar?q=Tianlin+Shi%2C+Andrej+Karpathy%2C+Linxi+Fan%2C+Jonathan+Hernandez%2C+and+Percy+Liang%2E+World+of+bits%3A+An+open%2Ddomain+platform+for+web%2Dbased+agents%2E+In+Doina+Precup+and+Yee+Whye+Teh+%28eds%2E%29%2C+Proc%2E+of+the+34th+In)
49. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.[11366, 2023](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/).
50. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. [Alfred: A benchmark for interpreting grounded instructions for everyday tasks](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/), 2020.
51. Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan Jepson, and Sheila A. McIlraith. Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning, 2021. URL https://arxiv.org/abs/2106.00133. — [arXiv:2106.00133](https://arxiv.org/abs/2106.00133) [↗](https://arxiv.org/abs/2106.00133)
52. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukc… [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J%2E+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+van+den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+Sander+Dieleman%2C+Dominik+)
53. Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, and Zongqing Lu. Towards General Computer Control: A Multimodal Agent For Red Dead Redemption II As A Case Stu… — [arXiv:2403.03186](https://arxiv.org/abs/2403.03186) [find](https://scholar.google.com/scholar?q=Weihao+Tan%2C+Ziluo+Ding%2C+Wentao+Zhang%2C+Boyu+Li%2C+Bohan+Zhou%2C+Junpeng+Yue%2C+Haochong+Xia%2C+Jiechuan+Jiang%2C+Longtao+Zheng%2C+Xinrun+Xu%2C+Yifei+Bi%2C+Pengjie+Gu%2C+Xinrun+Wang%2C+Bo%CC%88rje+F%2E+Karlsson%2C+Bo+An%2C+and+Zongqi)
54. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. January 2018. [find](https://scholar.google.com/scholar?q=Yuval+Tassa%2C+Yotam+Doron%2C+Alistair+Muldal%2C+Tom+Erez%2C+Yazhe+Li%2C+Diego+de+Las+Casas%2C+David+Budden%2C+Abbas+Abdolmaleki%2C+Josh+Merel%2C+Andrew+Lefrancq%2C+Timothy+Lillicrap%2C+and+Martin+Riedmiller%2E+DeepMind+cont)
55. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+Le%CC%81onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Rame%CC%81%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+)
56. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
57. Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android, 2021. URL https://arxiv.org/abs/2105.13231. — [arXiv:2105.13231](https://arxiv.org/abs/2105.13231) [↗](https://arxiv.org/abs/2105.13231)
58. Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. Ugif: Ui grounded instruction following, 2022. URL https://arxiv.org/abs/2211.07615. — [arXiv:2211.07615](https://arxiv.org/abs/2211.07615) [↗](https://arxiv.org/abs/2211.07615)
59. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vez… [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Igor+Babuschkin%2C+Wojciech+M+Czarnecki%2C+Michae%CC%88l+Mathieu%2C+Andrew+Dudzik%2C+Junyoung+Chung%2C+David+H+Choi%2C+Richard+Powell%2C+Timo+Ewalds%2C+Petko+Georgiev%2C+Junhyuk+Oh%2C+Dan+Horgan%2C+Manuel+Kroiss%2C)
60. Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. In Proc. of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23. Association for Computing Machinery, 2023a. ISBN 9781450394215. doi: 10.1145/3544548.3580895. URL http… — [doi:10.1145/3544548.3580895](https://doi.org/10.1145/3544548.3580895) [find](https://scholar.google.com/scholar?q=Bryan+Wang%2C+Gang+Li%2C+and+Yang+Li%2E+Enabling+conversational+interaction+with+mobile+ui+using+large+language+models%2E+In+Proc%2E+of+the+2023+CHI+Conference+on+Human+Factors+in+Computing+Systems%2C+CHI+%E2%80%9923%2E+As)
61. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023b. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A23)
62. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. [Autogen: Enabling next-gen llm applications via multi-agent conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/) framework. 2023.
63. Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. [Os-copilot: Towards generalist computer agents with self-improvement](https://md.research.elamur.ai/free/en/papers/wu-2024-os-copilot-towards-generalist-computer/). arXiv preprint arXiv:2402.07456, 2024.
64. Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. [Openagents: An open platform for language agents in the wild](https://md.research.elamur.ai/free/en/papers/xie-2023-openagents-open-platform-language-agents/), 2023.
65. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. [Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/). arXiv preprint arXiv:2404.07972, 2024.
66. Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within a complex android environment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6061– 6072, 2024. [find](https://scholar.google.com/scholar?q=Mingzhe+Xing%2C+Rongkai+Zhang%2C+Hui+Xue%2C+Qi+Chen%2C+Fan+Yang%2C+and+Zhen+Xiao%2E+Understanding+the+weakness+of+large+language+model+agents+within+a+complex+android+environment%2E+In+Proceedings+of+the+30th+ACM+S)
67. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. [On the tool manipulation capability of open-source large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-tool-manipulation-capability-open-source/). May 2023.
68. An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. GPT-4V in wonderland: Large multimodal models for Zero-Shot smartphone GUI navigation. November 2023. [find](https://scholar.google.com/scholar?q=An+Yan%2C+Zhengyuan+Yang%2C+Wanrong+Zhu%2C+Kevin+Lin%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Jianwei+Yang%2C+Yiwu+Zhong%2C+Julian+McAuley%2C+Jianfeng+Gao%2C+Zicheng+Liu%2C+and+Lijuan+Wang%2E+GPT%2D4V+in+wonderland%3A+Large+multimodal+m)
69. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. [Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v](https://md.research.elamur.ai/free/en/papers/yang-2023-set-mark-prompting-unleashes-extraordinary/). arXiv preprint arXiv:2310.11441, 2023a.
70. Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. [Appagent: Multimodal agents as smartphone users](https://md.research.elamur.ai/free/en/papers/zhang-2023-appagent-multimodal-agents-smartphone-users/). arXiv preprint arXiv:2312.13771, 2023b.
71. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [ReAct: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). October 2022.
72. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. [Webshop: Towards scalable real-world web interaction with grounded language agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/), 2023.
73. Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms, 2024. URL https://arxiv.org/abs/2404.05719. — [arXiv:2404.05719](https://arxiv.org/abs/2404.05719) [↗](https://arxiv.org/abs/2404.05719)
74. Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. [Ufo: A ui-focused agent for windows os interaction](https://md.research.elamur.ai/free/en/papers/zhang-2025-ufo-ui-focused-agent-windows/). arXiv preprint arXiv:2402.07939, 2024a.
75. Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, and Kai Yu. Mobile-env: Building qualified evaluation benchmarks for llm-gui interaction, 2024b. URL https://arxiv.org/abs/2305. 08144. [↗](https://arxiv.org/abs/2305)
76. Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713, 2024c. [find](https://scholar.google.com/scholar?q=Jiwen+Zhang%2C+Jihao+Wu%2C+Yihua+Teng%2C+Minghui+Liao%2C+Nuo+Xu%2C+Xiao+Xiao%2C+Zhongyu+Wei%2C+and+Duyu+Tang%2E+Android+in+the+zoo%3A+Chain%2Dof%2Daction%2Dthought+for+gui+agents%2E+arXiv+preprint+arXiv%3A2403%2E02713%2C+2024c%2E)
77. Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task automation, 2024d. URL https://arxiv.org/abs/2404.16054. — [arXiv:2404.16054](https://arxiv.org/abs/2404.16054) [↗](https://arxiv.org/abs/2404.16054)
78. Zhuosheng Zhang and Aston Zhang. [You only look at screens: Multimodal chain-of-action agents](https://md.research.elamur.ai/free/en/papers/zhang-2024-you-only-look-screens-multimodal/), 2023.
79. Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, et al. Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents. arXiv preprint arXiv:2311.11797, 2023. — [arXiv:2311.11797](https://arxiv.org/abs/2311.11797) [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Yao+Yao%2C+Aston+Zhang%2C+Xiangru+Tang%2C+Xinbei+Ma%2C+Zhiwei+He%2C+Yiming+Wang%2C+Mark+Gerstein%2C+Rui+Wang%2C+Gongshen+Liu%2C+et+al%2E+Igniting+language+intelligence%3A+The+hitchhiker%E2%80%99s+guide+from+chain%2D)
80. Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. MMInA: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.09992, 2024e. [find](https://scholar.google.com/scholar?q=Ziniu+Zhang%2C+Shulin+Tian%2C+Liangyu+Chen%2C+and+Ziwei+Liu%2E+MMInA%3A+Benchmarking+multihop+multimodal+internet+agents%2E+arXiv+preprint+arXiv%3A2404%2E09992%2C+2024e%2E)
81. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. [Gpt-4v(ision) is a generalist web agent, if grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). arXiv preprint arXiv:2401.01614, 2024a.
82. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. [Judging LLM-as-a-Judge with MT-Bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). June 2023.
83. Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents, 2024b. URL https://arxiv.org/abs/2403.17918. — [arXiv:2403.17918](https://arxiv.org/abs/2403.17918) [↗](https://arxiv.org/abs/2403.17918)
84. Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control, 2024c. [find](https://scholar.google.com/scholar?q=Longtao+Zheng%2C+Rundong+Wang%2C+Xinrun+Wang%2C+and+Bo+An%2E+Synapse%3A+Trajectory%2Das%2Dexemplar+prompting+with+memory+for+computer+control%2C+2024c%2E)
85. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/), 2023.

*Updated 2 November 2026*