---
title: "OS-ATLAS: A Foundation Action Model for Generalist GUI Agents"
description: "GUI agents that operate directly on screenshots usually depend on proprietary vision-language… 45 references, cited by 67 papers here."
authors: ["Zhiyong Wu", "Zhenyu Wu", "Fangzhi Xu", "Yian Wang", "Qiushi Sun", "Chengyou Jia", "Kanzhi Cheng", "Zichen Ding", "Liheng Chen", "Paul Pu Liang", "Yu Qiao"]
year: 2024
venue: "arXiv:2410.23218"
source: "https://arxiv.org/abs/2410.23218"
doi: "10.48550/arxiv.2410.23218"
arxiv: "2410.23218"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2024-os-atlas-foundation-action-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2024-os-atlas-foundation-action-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

# OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

**Authors:** [Zhiyong Wu](https://md.research.elamur.ai/free/en/authors/zhiyong-wu/), [Zhenyu Wu](https://md.research.elamur.ai/free/en/authors/zhenyu-wu/), [Fangzhi Xu](https://md.research.elamur.ai/free/en/authors/fangzhi-xu/), [Yian Wang](https://md.research.elamur.ai/free/en/authors/yian-wang/), [Qiushi Sun](https://md.research.elamur.ai/free/en/authors/qiushi-sun/), [Chengyou Jia](https://md.research.elamur.ai/free/en/authors/chengyou-jia/), [Kanzhi Cheng](https://md.research.elamur.ai/free/en/authors/kanzhi-cheng/), [Zichen Ding](https://md.research.elamur.ai/free/en/authors/zichen-ding/), [Liheng Chen](https://md.research.elamur.ai/free/en/authors/liheng-chen/), [Paul Pu Liang](https://md.research.elamur.ai/free/en/authors/paul-pu-liang/) and [Yu Qiao](https://md.research.elamur.ai/free/en/authors/yu-qiao/) · **Year:** 2024 · **Venue:** arXiv:2410.23218 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.23218) · [PDF](https://arxiv.org/pdf/2410.23218) · [DOI 10.48550/arxiv.2410.23218](https://doi.org/10.48550/arxiv.2410.23218) · [arXiv 2410.23218](https://arxiv.org/abs/2410.23218) · [OpenAlex](https://openalex.org/W4404343203)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/wu-2024-os-atlas-foundation-action-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
GUI agents that operate directly on screenshots usually depend on proprietary vision-language models such as GPT-4o. Open-source VLMs lag behind at GUI grounding, which means mapping an instruction to a screen element. They also struggle with unfamiliar, out-of-distribution interfaces. The authors give two reasons for this gap: VLMs rarely see GUI screenshots in pretraining, and agent datasets name equivalent actions differently (e.g., "tap" vs. "click").

To address this, they built an open toolkit that automatically generates grounding data on Windows, Linux, MacOS, Android and the web. With it they assembled about 2.3 million screenshots containing over 13 million annotated elements. Training runs in two stages. Grounding pre-training produces OS-Atlas-Base, and multi-task action fine-tuning over a unified action space then removes naming conflicts.

The model can work as a grounding module behind a planner, as a standalone step-level action model, or as a base for task-specific fine-tuning. On six benchmarks covering mobile, desktop and web, it beats earlier state-of-the-art models. This makes it a credible open alternative to commercial VLMs.

## Key points
- The grounding corpus totals 13,582,210 elements and 2,240,717 screenshots and includes desktop data absent from earlier collections.
  - Synthetic sources include FineWeb-filtered web pages, Windows, Linux and MacOS desktops, and Pixel6 mobile.
  - Public sources include SeeClick, AMEX and UIbert.
  - Instruction-grounding data comes from GPT-4o annotations of Mind2Web, AITZ and AMEX, plus AndroidControl and Wave-UI.
- The authors re-annotated 11.32% of the ScreenSpot samples, which they found to be incorrect, and released the corrected benchmark as ScreenSpot-V2.
- Two model sizes are available: a 4B model built on InternVL-2 and a 7B model built on Qwen2-VL.
- Average grounding accuracy on ScreenSpot-V2:

  | Setting | OS-Atlas-Base-7B | SeeClick |
  |---|---|---|
  | Standalone | 84.12 | 55.09 |
  | With GPT-4o as planner | 87.11 | 63.60 |

- Agentic evaluation uses GUI-Act-Web, OmniAct, AndroidControl (Low/High) and GUI-Odyssey splits. A click counts as correct if it lands within 14% of the screen width from the target.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,166 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 119 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 160 works

## Cite

Cite as: Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang and Yu Qiao. 2024. OS-ATLAS: A Foundation Action Model for Generalist GUI Agents. arXiv:2410.23218.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wu2024os,
  title = {{OS-ATLAS: A Foundation Action Model for Generalist GUI Agents}},
  author = {Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang and Yu Qiao},
  year = {2024},
  doi = {10.48550/arxiv.2410.23218},
  eprint = {2410.23218},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.23218}
}
```

- [← previous On the Effects of Data Scale on UI Control Agents](https://md.research.elamur.ai/free/en/papers/li-2024-effects-data-scale-ui-control/)
- [next → Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://md.research.elamur.ai/free/en/papers/wei-2024-assessing-brittleness-safety-alignment-pruning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

45 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023. URL [https://www.adept.ai/](https://www.adept.ai/) blog/fuyu-8b. [↗](https://www.adept.ai/)
2. Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407.17490, 2024. [find](https://scholar.google.com/scholar?q=Yuxiang+Chai%2C+Siyuan+Huang%2C+Yazhe+Niu%2C+Han+Xiao%2C+Liang+Liu%2C+Dingyu+Zhang%2C+Peng+Gao%2C+Shuai+Ren%2C+and+Hongsheng+Li%2E+Amex%3A+Android+multi%2Dannotation+expo+dataset+for+mobile+gui+agents%2E+arXiv+preprint+arXiv)
3. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile GUI agents. CoRR, abs/2406.11317, 2024a. doi: 10.48550/ARXIV.2406.1131… — [doi:10.48550/arxiv.2406.11317](https://doi.org/10.48550/arxiv.2406.11317) [find](https://scholar.google.com/scholar?q=Wentong+Chen%2C+Junbo+Cui%2C+Jinyi+Hu%2C+Yujia+Qin%2C+Junjie+Fang%2C+Yue+Zhao%2C+Chongyi+Wang%2C+Jun+Liu%2C+Guirong+Chen%2C+Yupeng+Huo%2C+Yuan+Yao%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Guicourse%3A+From+general+vision)
4. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317, 2024b. — [arXiv:2406.11317](https://arxiv.org/abs/2406.11317) [find](https://scholar.google.com/scholar?q=Wentong+Chen%2C+Junbo+Cui%2C+Jinyi+Hu%2C+Yujia+Qin%2C+Junjie+Fang%2C+Yue+Zhao%2C+Chongyi+Wang%2C+Jun+Liu%2C+Guirong+Chen%2C+Yupeng+Huo%2C+et+al%2E+Guicourse%3A+From+general+vision+language+models+to+versatile+gui+agents%2E+arX)
5. Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. [Agent-flan: Designing data and methods of effective agent tuning for large language models](https://md.research.elamur.ai/free/en/papers/chen-2024-agent-flan-designing-data-methods/). arXiv preprint arXiv:2403.12881, 2024c.
6. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. [How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/). arXiv preprint arXiv:2404.16821, 2024d.
7. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. [Seeclick: Harnessing gui grounding for advanced visual gui agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). arXiv preprint arXiv:2401.10935, 2024.
8. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). arXiv preprint arXiv:2306.06070, 2023a.
9. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=kiYqbO3wqw. [↗](https://openreview.net/forum?)
10. Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. Agent ai: Surveying the horizons of multimodal interaction, 2024. URL https://arxiv.org/abs/2401.03568. — [arXiv:2401.03568](https://arxiv.org/abs/2401.03568) [↗](https://arxiv.org/abs/2401.03568)
11. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. [Navigating the digital world as humans do: Universal visual grounding for gui agents](https://md.research.elamur.ai/free/en/papers/gou-2024-navigating-digital-world-humans-universal/). arXiv preprint arXiv:2410.05243, 2024.
12. Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby B. Lee, and Jindong Chen. Actionbert: Leveraging user actions for semantic understanding of user interfaces. In AAAI Conference on Artificial Intelligence, 2020. URL [https://api.semanticscholar.or…](https://api.semanticscholar.or…) [↗](https://api.semanticscholar.or…)
13. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. [MetaGPT: Meta programming for a multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). In The Twelft…
14. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281–14290, 2024b.
15. Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. CoRR, abs/2402.17553, 2024. doi: 10.48550/ARXIV. 2402.17553. URL https://do… [find](https://scholar.google.com/scholar?q=Raghav+Kapoor%2C+Yash+Parag+Butala%2C+Melisa+Russak%2C+Jing+Yu+Koh%2C+Kiran+Kamble%2C+Waseem+AlShikh%2C+and+Ruslan+Salakhutdinov%2E+Omniact%3A+A+dataset+and+benchmark+for+enabling+multimodal+generalist+autonomous+age)
16. Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. [Pix2struct: Screenshot parsing as pretraining for visual language understanding](https://md.research.elamur.ai/free/en/papers/lee-2022-pix2struct-screenshot-parsing-pretraining-visual/). ArXiv, abs/2210.03347, 2022. URL [https://api.semanticschol…](https://api.semanticschol…) [↗](https://api.semanticschol…)
17. Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.[03679, 2024](https://md.research.elamur.ai/free/en/papers/li-2024-effects-data-scale-ui-control/).
18. Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. [Mapping natural language instructions to mobile UI action sequences](https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/). In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.…
19. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2C+2024%2E)
20. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. [Grounding dino: Marrying dino with grounded pre-training for open-set object detection](https://md.research.elamur.ai/free/en/papers/liu-2023-grounding-dino-marrying-dino-grounded/). arXiv preprint arXiv:2303.05499, 2023.
21. Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. GUI odyssey: A comprehensive dataset for cross-app GUI navigation on mobile devices. CoRR, abs/2406.08451, 2024a. doi: 10.48550/ARXIV.2406.08451. URL https://doi.org/10.48550… — [doi:10.48550/arxiv.2406.08451](https://doi.org/10.48550/arxiv.2406.08451) [find](https://scholar.google.com/scholar?q=Quanfeng+Lu%2C+Wenqi+Shao%2C+Zitao+Liu%2C+Fanqing+Meng%2C+Boxuan+Li%2C+Botong+Chen%2C+Siyuan+Huang%2C+Kaipeng+Zhang%2C+Yu+Qiao%2C+and+Ping+Luo%2E+GUI+odyssey%3A+A+comprehensive+dataset+for+cross%2Dapp+GUI+navigation+on+mobil)
22. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203, 2024b. — [arXiv:2408.00203](https://arxiv.org/abs/2408.00203) [find](https://scholar.google.com/scholar?q=Yadong+Lu%2C+Jianwei+Yang%2C+Yelong+Shen%2C+and+Ahmed+Awadallah%2E+Omniparser+for+pure+vision+based+gui+agent%2E+arXiv+preprint+arXiv%3A2408%2E00203%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2408%2E00203%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E00203%29)
23. Guilherme Penedo, Hynek Kydlı́ček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/), 2024. URL https://arxiv.org/abs/2406.17557. [↗](https://arxiv.org/abs/2406.17557)
24. Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. [find](https://scholar.google.com/scholar?q=Theodore+R+Sumers%2C+Shunyu+Yao%2C+Karthik+Narasimhan%2C+and+Thomas+L+Griffiths%2E+Cognitive+architectures+for+language+agents%2E+arXiv+preprint+arXiv%3A2309%2E02427%2C+2023%2E)
25. Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. arXiv preprint arXiv:2310.00280, 2023. [find](https://scholar.google.com/scholar?q=Qiushi+Sun%2C+Zhangyue+Yin%2C+Xiang+Li%2C+Zhiyong+Wu%2C+Xipeng+Qiu%2C+and+Lingpeng+Kong%2E+Corex%3A+Pushing+the+boundaries+of+complex+reasoning+through+multi%2Dmodel+collaboration%2E+arXiv+preprint+arXiv%3A2310%2E00280%2C+20)
26. Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403.14734, 2024. — [arXiv:2403.14734](https://arxiv.org/abs/2403.14734) [find](https://scholar.google.com/scholar?q=Qiushi+Sun%2C+Zhirui+Chen%2C+Fangzhi+Xu%2C+Kanzhi+Cheng%2C+Chang+Ma%2C+Zhangyue+Yin%2C+Jianing+Wang%2C+Chengcheng+Han%2C+Renyu+Zhu%2C+Shuai+Yuan%2C+et+al%2E+A+survey+of+neural+code+intelligence%3A+Paradigms%2C+advances+and+bey)
27. Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A reinforcement learning platform for android. abs/2105.13231, 2021. URL http://arxiv.org/abs/2105.13231. — [arXiv:2105.13231](https://arxiv.org/abs/2105.13231) [↗](http://arxiv.org/abs/2105.13231)
28. Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. The 34th Annual ACM Symposium on User Interface Software and Technology, 2021. URL [https://api.semanticscholar.org/CorpusID:236957064](https://api.semanticscholar.org/CorpusID:236957064). [↗](https://api.semanticscholar.org/CorpusID:236957064)
29. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. [A survey on large language model based autonomous agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/). http://arxiv.org/abs/2308.11432, 2023. [↗](http://arxiv.org/abs/2308.11432)
30. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world a… — [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
31. WebAIM. Webaim: The webaim million - the 2024 report on the accessibility of the top 1,000,000 home pages, 2024. URL [https://webaim.org/projects/million/](https://webaim.org/projects/million/). Accessed on September 30, 2024. [↗](https://webaim.org/projects/million/)
32. Lilian Weng. Llm-powered autonomous agents. lilianweng.github.io, Jun 2023. URL [https://lilianweng.github.io/posts/2023-06-23-agent/](https://lilianweng.github.io/posts/2023-06-23-agent/). [↗](https://lilianweng.github.io/posts/2023-06-23-agent/)
33. Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. [Os-copilot: Towards generalist computer agents with self-improvement](https://md.research.elamur.ai/free/en/papers/wu-2024-os-copilot-towards-generalist-computer/). arXiv preprint arXiv:2402.07456, 2024.
34. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. [Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/). arXiv preprint arXiv:2404.07972, 2024.
35. Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. [Interactive evolution: A neural-symbolic self-training framework for large language models](https://md.research.elamur.ai/free/en/papers/xu-2025-interactive-evolution-neural-symbolic-self/). arXiv preprint arXiv:2406.11736, 2024.
36. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. [Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v](https://md.research.elamur.ai/free/en/papers/yang-2023-set-mark-prompting-unleashes-extraordinary/). arXiv preprint arXiv:2310.11441, 2023.
37. Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source llms. arXiv preprint arXiv:2311.05657, 2023. [find](https://scholar.google.com/scholar?q=Da+Yin%2C+Faeze+Brahman%2C+Abhilasha+Ravichander%2C+Khyathi+Chandu%2C+Kai%2DWei+Chang%2C+Yejin+Choi%2C+and+Bill+Yuchen+Lin%2E+Lumos%3A+Learning+agents+with+unified+data%2C+modular+design%2C+and+open%2Dsource+llms%2E+arXiv+prep)
38. Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv preprint arXiv:2404.05719, 2024. — [arXiv:2404.05719](https://arxiv.org/abs/2404.05719) [find](https://scholar.google.com/scholar?q=Keen+You%2C+Haotian+Zhang%2C+Eldon+Schoop%2C+Floris+Weers%2C+Amanda+Swearngin%2C+Jeffrey+Nichols%2C+Yinfei+Yang%2C+and+Zhe+Gan%2E+Ferret%2Dui%3A+Grounded+mobile+ui+understanding+with+multimodal+llms%2E+arXiv+preprint+arXiv)
39. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. [Agenttuning: Enabling generalized agent abilities for llms](https://md.research.elamur.ai/free/en/papers/zeng-2023-agenttuning-enabling-generalized-agent/). arXiv preprint arXiv:2310.12823, 2023.
40. Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. [Ufo: A ui-focused agent for windows os interaction](https://md.research.elamur.ai/free/en/papers/zhang-2025-ufo-ui-focused-agent-windows/). arXiv preprint arXiv:2402.07939, 2024a.
41. Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint arXiv:2402.15506, 2024b. — [arXiv:2402.15506](https://arxiv.org/abs/2402.15506) [find](https://scholar.google.com/scholar?q=Jianguo+Zhang%2C+Tian+Lan%2C+Rithesh+Murthy%2C+Zhiwei+Liu%2C+Weiran+Yao%2C+Juntao+Tan%2C+Thai+Hoang%2C+Liangwei+Yang%2C+Yihao+Feng%2C+Zuxin+Liu%2C+et+al%2E+Agentohana%3A+Design+unified+data+and+training+pipeline+for+effectiv)
42. Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. [xlam: A family of large action models to empower ai agent systems](https://md.research.elamur.ai/free/en/papers/zhang-2025-xlam-family-large-action-models/). arXiv preprint arXiv:2409.03215, 2024c.
43. Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents, 2024d. [find](https://scholar.google.com/scholar?q=Jiwen+Zhang%2C+Jihao+Wu%2C+Yihua+Teng%2C+Minghui+Liao%2C+Nuo+Xu%2C+Xiao+Xiao%2C+Zhongyu+Wei%2C+and+Duyu+Tang%2E+Android+in+the+zoo%3A+Chain%2Dof%2Daction%2Dthought+for+gui+agents%2C+2024d%2E)
44. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. [Gpt-4v(ision) is a generalist web agent, if grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). In Forty-first International Conference on Machine Learning, 2024a. URL [https://openreview.net/forum?id=piecKJ2DlB](https://openreview.net/forum?id=piecKJ2DlB). [↗](https://openreview.net/forum?id=piecKJ2DlB)
45. Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents. arXiv preprint arXiv:2403.17918, 2024b. — [arXiv:2403.17918](https://arxiv.org/abs/2403.17918) [find](https://scholar.google.com/scholar?q=Longtao+Zheng%2C+Zhiyuan+Huang%2C+Zhenghai+Xue%2C+Xinrun+Wang%2C+Bo+An%2C+and+Shuicheng+Yan%2E+Agentstudio%3A+A+toolkit+for+building+general+virtual+agents%2E+arXiv+preprint+arXiv%3A2403%2E17918%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2403%2E179)

*Updated 2 November 2026*