---
title: "You Only Look at Screens: Multimodal Chain-of-Action Agents"
description: "This creates long inputs, slow inference, parsing errors that carry forward, and dependence on APIs… 54 references, cited by 52 papers here."
authors: ["Zhuosheng Zhang", "Aston Zhang"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.186/"
doi: "10.18653/v1/2024.findings-acl.186"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-you-only-look-screens-multimodal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-you-only-look-screens-multimodal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / You Only Look at Screens: Multimodal Chain-of-Action Agents

# You Only Look at Screens: Multimodal Chain-of-Action Agents

**Authors:** [Zhuosheng Zhang](https://md.research.elamur.ai/free/en/authors/zhuosheng-zhang/) and [Aston Zhang](https://md.research.elamur.ai/free/en/authors/aston-zhang/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.186/) · [PDF](https://aclanthology.org/2024.findings-acl.186.pdf) · [DOI 10.18653/v1/2024.findings-acl.186](https://doi.org/10.18653/v1/2024.findings-acl.186) · [OpenAlex](https://openalex.org/W4402670435)

**Citations:** 28 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/zhang-2024-you-only-look-screens-multimodal/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most GUI agents built on large language models work in a "sandbox". External tools such as OCR and icon detectors turn the screen into text like HTML, and app-specific APIs carry out the predicted actions. This creates long inputs, slow inference, parsing errors that carry forward, and dependence on APIs that third-party apps often do not expose.

The authors propose Auto-GUI, a multimodal agent that reads raw screenshots and outputs directly executable actions: action type, touch and lift coordinates, and typed text. A frozen vision encoder (BLIP-2 by default) supplies image features. These are fused with a language encoder built on FLAN-Alpaca weights, and a decoder generates the output. The key addition is a chain-of-action mechanism. The model receives its previous actions as input and first predicts a plan of upcoming actions, then decides the current step.

On the AITW device-control benchmark, Auto-GUI reaches state-of-the-art results while predicting each action in under a second. This suggests that compact end-to-end multimodal models can make practical, API-free screen agents.

## Key points
- Evaluation uses AITW, which has about 30K unique instructions covering app operation, web search and web shopping. Auto-GUI reaches roughly 90% action-type accuracy and about 74% overall action success.
- Model sizes are base (220M) and large (770M). Growing the model gave only small gains compared with chain-of-action and coordinate normalization, while the choice of vision features (BLIP-2 over CLIP) mattered more.
- Auto-GUI large needs about 0.59 s per inference and 8.2 GB of peak GPU memory, and the base model needs 0.19 s and 4.6 GB. Llama 2 needs about 8.5 s and 49.7 GB.
- On the General test set, a ChatGPT in-context-learning baseline reaches only 5.93% overall success, compared with 68.24% for Auto-GUI.
- Code is available at github.com/cooelf/Auto-GUI. Stated limitations are that experiments cover only AITW and that very large models were not explored.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 63 works · 15,047 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 46 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 25 works · [⚙️ Industrial and Manufacturing Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/industrial-and-manufacturing-engineering/) · 4 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Zhuosheng Zhang and Aston Zhang. 2024. You Only Look at Screens: Multimodal Chain-of-Action Agents. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024you,
  title = {{You Only Look at Screens: Multimodal Chain-of-Action Agents}},
  author = {Zhuosheng Zhang and Aston Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.186},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.186}
}
```

- [← previous Do Membership Inference Attacks Work on Large Language Models?](https://md.research.elamur.ai/free/en/papers/duan-2024-membership-inference-attacks-work-large/)
- [next → Generating Benchmarks for Factuality Evaluation of Language Models](https://md.research.elamur.ai/free/en/papers/muhlgay-2024-generating-benchmarks-factuality-evaluation-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

54 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adept. 2022. Act-1: Transformer for actions. [https://www.adept.ai/act](https://www.adept.ai/act). [↗](https://www.adept.ai/act)
2. Aristotle. Physics 184a10–21. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. 2023. Introducing our multimodal models: Fuyu-8b. [find](https://scholar.google.com/scholar?q=Aristotle%2E+Physics+184a10%E2%80%9321%2E+Rohan+Bavishi%2C+Erich+Elsen%2C+Curtis+Hawthorne%2C+Maxwell+Nye%2C+Augustus+Odena%2C+Arushi+Somani%2C+and+Sag%CC%86nak+Tas%CC%A7%C4%B1rlar%2E+2023%2E+Introducing+our+multimodal+models%3A+Fuyu%2D8b%2E)
3. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. [Seeclick: Harnessing gui grounding for advanced visual gui agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). ArXiv preprint, abs/2401.10935.
5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [https://vicuna.lmsys.org](https://vicuna.lmsys.org). [↗](https://vicuna.lmsys.org)
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). ArXiv preprint, abs/2204.02311.
7. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). ArXiv preprint, abs/2210.11416.
8. Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. 2023. Assistgui: Task-oriented desktop graphical user interface automation. ArXiv preprint, abs/2312.13108. — [arXiv:2312.13108](https://arxiv.org/abs/2312.13108) [find](https://scholar.google.com/scholar?q=Difei+Gao%2C+Lei+Ji%2C+Zechen+Bai%2C+Mingyu+Ouyang%2C+Peiran+Li%2C+Dongxing+Mao%2C+Qinchen+Wu%2C+Weichen+Zhang%2C+Peiyi+Wang%2C+Xiangwu+Guo%2C+et+al%2E+2023%2E+Assistgui%3A+Task%2Doriented+desktop+graphical+user+interface+automa)
9. Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. ArXiv preprint, abs/2307.12856. [find](https://scholar.google.com/scholar?q=Izzeddin+Gur%2C+Hiroki+Furuta%2C+Austin+Huang%2C+Mustafa+Safdari%2C+Yutaka+Matsuo%2C+Douglas+Eck%2C+and+Aleksandra+Faust%2E+2023%2E+A+real%2Dworld+webagent+with+planning%2C+long+context+understanding%2C+and+program+synthes)
10. James Hendler. 1999. Is there an intelligent agent in your future? Nature, 11. [find](https://scholar.google.com/scholar?q=James+Hendler%2E+1999%2E+Is+there+an+intelligent+agent+in+your+future%3F+Nature%2C+11%2E)
11. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. 2023a. Metagpt: Meta programming for multi-agent collaborative framework. [find](https://scholar.google.com/scholar?q=Sirui+Hong%2C+Xiawu+Zheng%2C+Jonathan+Chen%2C+Yuheng+Cheng%2C+Jinlin+Wang%2C+Ceyao+Zhang%2C+Zili+Wang%2C+Steven+Ka+Shing+Yau%2C+Zijuan+Lin%2C+Liyang+Zhou%2C+Chenyu+Ran%2C+Lingfeng+Xiao%2C+and+Chenglin+Wu%2E+2023a%2E+Metagpt%3A+Met)
12. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. 2023b. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). ArXiv preprint, abs/2312.08914.
13. Terence Irwin. 1989. Aristotle’s first principles. Clarendon Press. [find](https://scholar.google.com/scholar?q=Terence+Irwin%2E+1989%2E+Aristotle%E2%80%99s+first+principles%2E+Clarendon+Press%2E)
14. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. [Visualwebarena: Evaluating multimodal agents on realistic visual web tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/). ArXiv preprint, abs/2401.13649.
15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). ArXiv preprint, abs/2205.11916.
16. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). ArXiv preprint, abs/2301.12597.
17. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). ArXiv preprint, abs/2308.03688.
18. Pattie Maes. 1995. Agents that reduce work and information overload. In Readings in human–computer interaction, pages 811–821. Elsevier. [find](https://scholar.google.com/scholar?q=Pattie+Maes%2E+1995%2E+Agents+that+reduce+work+and+information+overload%2E+In+Readings+in+human%E2%80%93computer+interaction%2C+pages+811%E2%80%93821%2E+Elsevier%2E)
19. Yohei Nakajima. 2023. Babyagi. [https://github.com/yoheinakajima/babyagi](https://github.com/yoheinakajima/babyagi). [↗](https://github.com/yoheinakajima/babyagi)
20. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2022. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). In Deep Learning for Code Workshop.
21. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
22. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. ArXiv preprint, abs/2304.03442. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C+O%E2%80%99Brien%2C+Carrie+J+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+2023%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+ArXiv+preprint%2C+abs%2F2304)
23. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the 38th Internatio…
24. July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR. [find](https://scholar.google.com/scholar?q=July+2021%2C+Virtual+Event%2C+volume+139+of+Proceedings+of+Machine+Learning+Research%2C+pages+8748%E2%80%938763%2E+PMLR%2E)
25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
26. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap. 2023. An-droidinthewild: A large-scale dataset for android device control. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Christopher+Rawles%2C+Alice+Li%2C+Daniel+Rodriguez%2C+Oriana+Riva%2C+and+Timothy+P+Lillicrap%2E+2023%2E+An%2Ddroidinthewild%3A+A+large%2Dscale+dataset+for+android+device+control%2E+In+Thirty%2Dseventh+Conference+on+Neural+)
27. Reworkd. 2023. [https://github.com/reworkd/AgentGPT](https://github.com/reworkd/AgentGPT). [↗](https://github.com/reworkd/AgentGPT)
28. Toran Bruce Richards. 2023. Auto-gpt: An autonomous gpt-4 experiment. [https://github.com/Significant-Gravitas/Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT). [↗](https://github.com/Significant-Gravitas/Auto-GPT)
29. [Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonatha…](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
30. John R Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge university press. [find](https://scholar.google.com/scholar?q=John+R+Searle%2E+1969%2E+Speech+acts%3A+An+essay+in+the+philosophy+of+language%2C+volume+626%2E+Cambridge+university+press%2E)
31. Yunpeng Song, Yiheng Bian, Yongtao Tang, and Zhongmin Cai. 2023. Navigating interfaces with ai for enhanced user interaction. ArXiv preprint, abs/2312.11190. — [arXiv:2312.11190](https://arxiv.org/abs/2312.11190) [find](https://scholar.google.com/scholar?q=Yunpeng+Song%2C+Yiheng+Bian%2C+Yongtao+Tang%2C+and+Zhongmin+Cai%2E+2023%2E+Navigating+interfaces+with+ai+for+enhanced+user+interaction%2E+ArXiv+preprint%2C+abs%2F2312%2E11190%2E+%E2%80%94+%5BarXiv%3A2312%2E11190%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
32. Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. 2023. Cognitive architectures for language agents. ArXiv preprint, abs/2309.02427. [find](https://scholar.google.com/scholar?q=Theodore+Sumers%2C+Shunyu+Yao%2C+Karthik+Narasimhan%2C+and+Thomas+L+Griffiths%2E+2023%2E+Cognitive+architectures+for+language+agents%2E+ArXiv+preprint%2C+abs%2F2309%2E02427%2E)
33. Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. META-GUI: Towards multi-modal conversational agents on mobile GUI. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6699–6712, Abu Dhabi, United Arab Emirates. Association fo… [find](https://scholar.google.com/scholar?q=Liangtai+Sun%2C+Xingyu+Chen%2C+Lu+Chen%2C+Tianle+Dai%2C+Zichen+Zhu%2C+and+Kai+Yu%2E+2022%2E+META%2DGUI%3A+Towards+multi%2Dmodal+conversational+agents+on+mobile+GUI%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Meth)
34. Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James W. W. Stout. 2022. Towards better semantic understanding of mobile interfaces. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5636– 5650, Gyeon… [find](https://scholar.google.com/scholar?q=Srinivas+Sunkara%2C+Maria+Wang%2C+Lijuan+Liu%2C+Gilles+Baechler%2C+Yu%2DChung+Hsiao%2C+Jindong+Chen%2C+Abhanshu+Sharma%2C+and+James+W%2E+W%2E+Stout%2E+2022%2E+Towards+better+semantic+understanding+of+mobile+interfaces%2E+In+Pr)
35. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023a. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html. [↗](https://crfm.stanford)
36. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023b. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
39. Bryan Wang, Gang Li, and Yang Li. 2023a. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17. [find](https://scholar.google.com/scholar?q=Bryan+Wang%2C+Gang+Li%2C+and+Yang+Li%2E+2023a%2E+Enabling+conversational+interaction+with+mobile+ui+using+large+language+models%2E+In+Proceedings+of+the+2023+CHI+Conference+on+Human+Factors+in+Computing+Systems)
40. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023b. Voyager: An open-ended embodied agent with large language models. ArXiv preprint, abs/2305.16291. [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+2023b%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+ArXiv+preprint%2C+)
41. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023c. [A survey on large language model based autonomous agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/). ArXiv preprint, abs/2308.11432.
42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). ArXiv preprint, abs/2201.11903.
43. Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023. Empowering llm to use smartphone for intelligent task automation. ArXiv preprint, abs/2308.15272. — [arXiv:2308.15272](https://arxiv.org/abs/2308.15272) [find](https://scholar.google.com/scholar?q=Hao+Wen%2C+Yuanchun+Li%2C+Guohong+Liu%2C+Shanhui+Zhao%2C+Tao+Yu%2C+Toby+Jia%2DJun+Li%2C+Shiqi+Jiang%2C+Yunhao+Liu%2C+Yaqin+Zhang%2C+and+Yunxin+Liu%2E+2023%2E+Empowering+llm+to+use+smartphone+for+intelligent+task+automation%2E+)
44. Michael Wooldridge and Nicholas R Jennings. 1995. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115–152. [find](https://scholar.google.com/scholar?q=Michael+Wooldridge+and+Nicholas+R+Jennings%2E+1995%2E+Intelligent+agents%3A+Theory+and+practice%2E+The+knowledge+engineering+review%2C+10%282%29%3A115%E2%80%93152%2E)
45. Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. [Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation](https://md.research.elamur.ai/free/en/papers/wu-2021-good-misconceived-reasons-empirical-revisiting/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation…
46. Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. 2023. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. ArXiv preprint, abs/2305.17144. [find](https://scholar.google.com/scholar?q=Xizhou+Zhu%2C+Yuntao+Chen%2C+Hao+Tian%2C+Chenxin+Tao%2C+Weijie+Su%2C+Chenyu+Yang%2C+Gao+Huang%2C+Bin+Li%2C+Lewei+Lu%2C+Xiaogang+Wang%2C+et+al%2E+2023%2E+Ghost+in+the+minecraft%3A+Generally+capable+agents+for+open%2Dworld+envirom)
47. An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. 2023. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. ArXiv preprint, abs/2311.07562. — [arXiv:2311.07562](https://arxiv.org/abs/2311.07562) [find](https://scholar.google.com/scholar?q=An+Yan%2C+Zhengyuan+Yang%2C+Wanrong+Zhu%2C+Kevin+Lin%2C+Linjie+Li%2C+Jianfeng+Wang%2C+Jianwei+Yang%2C+Yiwu+Zhong%2C+Julian+McAuley%2C+Jianfeng+Gao%2C+et+al%2E+2023%2E+Gpt%2D4v+in+wonderland%3A+Large+multimodal+models+for+zero%2Dsh)
48. Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. Appagent: Multimodal agents as smartphone users. ArXiv preprint, abs/2312.13771. [find](https://scholar.google.com/scholar?q=Zhao+Yang%2C+Jiaxuan+Liu%2C+Yucheng+Han%2C+Xin+Chen%2C+Zebiao+Huang%2C+Bin+Fu%2C+and+Gang+Yu%2E+2023%2E+Appagent%3A+Multimodal+agents+as+smartphone+users%2E+ArXiv+preprint%2C+abs%2F2312%2E13771%2E)
49. Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021. Screen recognition: Creating accessibility metadata for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in… [find](https://scholar.google.com/scholar?q=Xiaoyi+Zhang%2C+Lilian+de+Greef%2C+Amanda+Swearngin%2C+Samuel+White%2C+Kyle+Murray%2C+Lisa+Yu%2C+Qi+Shan%2C+Jeffrey+Nichols%2C+Jason+Wu%2C+Chris+Fleizach%2C+et+al%2E+2021%2E+Screen+recognition%3A+Creating+accessibility+metadat)
50. Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal visual representation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Kehai+Chen%2C+Rui+Wang%2C+Masao+Utiyama%2C+Eiichiro+Sumita%2C+Zuchao+Li%2C+and+Hai+Zhao%2E+2020%2E+Neural+machine+translation+with+universal+visual+representation%2E+In+8th+International+Conference+o)
51. Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, and Michael Zeng. 2022. Task compass: Scaling multi-task pre-training with task prefix. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5671–5685, Abu Dhabi, United A… [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Shuohang+Wang%2C+Yichong+Xu%2C+Yuwei+Fang%2C+Wenhao+Yu%2C+Yang+Liu%2C+Hai+Zhao%2C+Chenguang+Zhu%2C+and+Michael+Zeng%2E+2022%2E+Task+compass%3A+Scaling+multi%2Dtask+pre%2Dtraining+with+task+prefix%2E+In+Finding)
52. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023a. [Automatic chain of thought prompting in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/). In The Eleventh International Conference on Learning Representations.
53. Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023b. [Multimodal chain-of-thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-multimodal-chain-thought-reasoning-language/). ArXiv preprint, abs/2302.00923.
54. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). ArXiv preprint, abs/2307.13854.

*Updated 24 October 2026*