---
title: "AppWorld — paper by Trivedi et al. (2024) — summary and ideas"
description: "Earlier tool-use benchmarks mostly test short, linear chains of API calls. 44 references, cited by 60 papers here."
authors: ["Harsh Trivedi", "Tushar Khot", "Mareike Hartmann", "Ruskin Manku", "Vinty Dong", "Edward Li", "Shashank Gupta", "Ashish Sabharwal", "Niranjan Balasubramanian"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.850/"
doi: "10.18653/v1/2024.acl-long.850"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/trivedi-2024-appworld-controllable-world-apps-people/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/trivedi-2024-appworld-controllable-world-apps-people/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

# AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

**Authors:** [Harsh Trivedi](https://md.research.elamur.ai/free/en/authors/harsh-trivedi/), [Tushar Khot](https://md.research.elamur.ai/free/en/authors/tushar-khot/), [Mareike Hartmann](https://md.research.elamur.ai/free/en/authors/mareike-hartmann/), Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, [Ashish Sabharwal](https://md.research.elamur.ai/free/en/authors/ashish-sabharwal/) and [Niranjan Balasubramanian](https://md.research.elamur.ai/free/en/authors/niranjan-balasubramanian/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.850/) · [PDF](https://aclanthology.org/2024.acl-long.850.pdf) · [DOI 10.18653/v1/2024.acl-long.850](https://doi.org/10.18653/v1/2024.acl-long.850) · [OpenAlex](https://openalex.org/W4402671485)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/trivedi-2024-appworld-controllable-world-apps-people/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Agents that automate everyday digital chores, such as paying roommates or picking a playlist, must use several apps and write code that adapts to what they observe along the way. Earlier tool-use benchmarks mostly test short, linear chains of API calls. They also grade outputs against a reference solution, which handles tasks with many valid solutions poorly.

The authors built AppWorld to address this. It has two parts:
- **AppWorld Engine** is a sandboxed, fully resettable simulator of 9 everyday apps (e.g., Gmail, Venmo, Amazon, Spotify) exposed through documented APIs. It is filled with the simulated digital lives of about 100 fictitious people.
- **AppWorld Benchmark** contains 750 tasks that require iterative, multi-app coding. Each task is graded with database-level unit tests. These tests accept any valid route to the goal and also flag unintended side effects.

Prompting methods such as ReAct and Plan-And-Execute were tested with GPT4 variants, LLaMA3 and DeepSeek-Coder. The strongest setup, GPT4o with ReAct, completes fewer than half of the normal tasks. This shows how far current LLM agents are from dependable autonomy in realistic, consequence-free test settings.

## Key points
- **Engine scale:** 9 apps, 457 APIs with 1470 arguments, and 101 DB tables, plus the helper apps ApiDocs and Supervisor. It is built on FastAPI, SQLite and SQLModel, and agents work through a Jupyter-style stateful shell.
- **Task complexity:** tasks average 1.8 apps (max 6), 9.5 APIs (max 26) and about 50 lines of solution code (max 134). Each task has on average 8 state-based tests (max 22).
- **Best results:** GPT4o with ReAct scores 48.8 on Test-Normal and 30.2 on Test-Challenge. GPT4 Trb, the next best, scores 32.7 and 17.5. Under the scenario-based robustness metric, GPT4o falls to 32.1 and 13.0.
- **Engineering effort:** the whole framework comprises over 100K lines of code, written by the authors over 14 months rather than by crowdworkers or LLMs.
- **Limitations:** the benchmark covers API-only interaction, single-agent tasks and a dataset too small for training. UI-based and multi-agent extensions are left to future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations
- [Saarland University](https://md.research.elamur.ai/free/en/universities/saarland-university/) — University, Saarbrücken, Germany · 102 works · 17,447 citations
- [Stony Brook University](https://md.research.elamur.ai/free/en/universities/stony-brook-university/) — University, Stony Brook, United States · 27 works · 2,057 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 56 works

## Cite

Cite as: Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal and Niranjan Balasubramanian. 2024. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{trivedi2024appworld,
  title = {{AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents}},
  author = {Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal and Niranjan Balasubramanian},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.850},
  url = {https://doi.org/10.18653/v1/2024.acl-long.850}
}
```

- [← previous MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://md.research.elamur.ai/free/en/papers/ahuja-2024-megaverse-benchmarking-large-language-models/)
- [next → INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge](https://md.research.elamur.ai/free/en/papers/romanou-2024-include-evaluating-multilingual-language-understanding/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

44 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, abs/2303.08774.
2. BIG bench authors. 2023. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Research. 16031 10
3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv, abs/2107.03374.
4. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. [SeeClick: Harnessing gui grounding for advanced visual gui agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). In ACL.
5. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. [Mind2Web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). In NeurIPS: Datasets and Benchmarks Track.
6. Nicholas Farn and Richard Shin. 2023. ToolTalk: Evaluating tool-usage in a conversational setting. arXiv, abs/2311.10775. — [arXiv:2311.10775](https://arxiv.org/abs/2311.10775) [find](https://scholar.google.com/scholar?q=Nicholas+Farn+and+Richard+Shin%2E+2023%2E+ToolTalk%3A+Evaluating+tool%2Dusage+in+a+conversational+setting%2E+arXiv%2C+abs%2F2311%2E10775%2E+%E2%80%94+%5BarXiv%3A2311%2E10775%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E10775%29)
7. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, N… — [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
8. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. DeepSeek-Coder: When the large language model meets programming–the rise of code intelligence. arXiv, abs/2401.14196. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y+Wu%2C+YK+Li%2C+et+al%2E+2024%2E+DeepSeek%2DCoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code)
9. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. [WebVoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). arXiv, abs/2401.13919.
10. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée La… [find](https://scholar.google.com/scholar?q=Albert+Qiaochu+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%E2%80%99elio+Rena)
11. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. [SWE-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In ICLR.
12. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. [VisualWebArena: Evaluating multimodal agents on realistic visual web tasks](https://md.research.elamur.ai/free/en/papers/koh-2024-visualwebarena-evaluating-multimodal-agents-realistic/). In ACL.
13. Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. 2024. Benchmarking mobile device control agents across diverse configurations. [find](https://scholar.google.com/scholar?q=Juyong+Lee%2C+Taywon+Min%2C+Minyong+An%2C+Changyeon+Kim%2C+and+Kimin+Lee%2E+2024%2E+Benchmarking+mobile+device+control+agents+across+diverse+configurations%2E)
14. Minghao Li, Feifan Song, Yu Bowen, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. [API-Bank: A comprehensive benchmark for tool-augmented LLMs](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/). In EMNLP.
15. Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. WebLINX: Real-world website navigation with multi-turn dialogue. arXiv, abs/2402.05930. [find](https://scholar.google.com/scholar?q=Xing+Han+L%C3%B9%2C+Zdene%CC%8Ck+Kasner%2C+and+Siva+Reddy%2E+2024%2E+WebLINX%3A+Real%2Dworld+website+navigation+with+multi%2Dturn+dialogue%2E+arXiv%2C+abs%2F2402%2E05930%2E)
16. Meta AI. 2024. Introducing Meta Llama 3: The most capable openly available LLM to date. Blog post: [https://ai.meta.com/blog/meta-llama-3](https://ai.meta.com/blog/meta-llama-3). [↗](https://ai.meta.com/blog/meta-llama-3)
17. OpenAI. 2023a. ChatGPT. Https://chat.openai.com. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+ChatGPT%2E+Https%3A%2F%2Fchat%2Eopenai%2Ecom%2E)
18. OpenAI. 2023b. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774. arXiv, Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive APIs. arXiv, abs/2305.15334.
19. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. [ToolLLM: Facilitating large language models to master 16000+ real…](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
20. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. 2024. AndroidWorld: A dynamic benchmarking environment f… [find](https://scholar.google.com/scholar?q=Christopher+Rawles%2C+Sarah+Clinckemaillie%2C+Yifan+Chang%2C+Jonathan+Waltz%2C+Gabrielle+Lau%2C+Marybeth+Fair%2C+Alice+Li%2C+William+Bishop%2C+Wei+Li%2C+Folawiyo+Campbell%2DAjala%2C+Daniel+Toyama%2C+Robert+Berry%2C+Divya+Tyama)
21. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P. Lillicrap. 2023. Android in the wild: A large-scale dataset for android device control. In NeurIPS: Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Christopher+Rawles%2C+Alice+Li%2C+Daniel+Rodriguez%2C+Oriana+Riva%2C+and+Timothy+P%2E+Lillicrap%2E+2023%2E+Android+in+the+wild%3A+A+large%2Dscale+dataset+for+android+device+control%2E+In+NeurIPS%3A+Datasets+and+Benchmarks+)
22. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2023. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022.
23. Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yue Ting Zhuang. 2023. TaskBench: Benchmarking large language models for task automation. arXiv, abs/2311.18760. — [arXiv:2311.18760](https://arxiv.org/abs/2311.18760) [find](https://scholar.google.com/scholar?q=Yongliang+Shen%2C+Kaitao+Song%2C+Xu+Tan%2C+Wenqi+Zhang%2C+Kan+Ren%2C+Siyu+Yuan%2C+Weiming+Lu%2C+Dongsheng+Li%2C+and+Yue+Ting+Zhuang%2E+2023%2E+TaskBench%3A+Benchmarking+large+language+models+for+task+automation%2E+arXiv%2C+abs)
24. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In NeurIPS.
25. Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. 2023. [RestGPT: Connecting large language models with real-world RESTful APIs](https://md.research.elamur.ai/free/en/papers/song-2023-restgpt-connecting-large-language-models/). arXiv, abs/2306.06624. 16032 11
26. Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. 2022. META-GUI: Towards multi-modal conversational agents on mobile GUI. In EMNLP. [find](https://scholar.google.com/scholar?q=Liangtai+Sun%2C+Xingyu+Chen%2C+Lu+Chen%2C+Tianle+Dai%2C+Zichen+Zhu%2C+and+Kai+Yu%2E+2022%2E+META%2DGUI%3A+Towards+multi%2Dmodal+conversational+agents+on+mobile+GUI%2E+In+EMNLP%2E)
27. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. [WebShop: Towards scalable real-world web interaction with grounded language agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/). In NeurIPS.
28. Simeng Sun, Yang Liu, Shuohang Wang, Dan Iter, Chenguang Zhu, and Mohit Iyyer. 2024. PEARL: Prompting large language models to plan and execute actions over long documents. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Simeng+Sun%2C+Yang+Liu%2C+Shuohang+Wang%2C+Dan+Iter%2C+Chenguang+Zhu%2C+and+Mohit+Iyyer%2E+2024%2E+PEARL%3A+Prompting+large+language+models+to+plan+and+execute+actions+over+long+documents%2E+In+Proceedings+of+the+18th+)
29. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. [ReAct: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In ICLR.
30. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. [ToolAlpaca: Generalized tool learning for language models with 3000 simulated cases](https://md.research.elamur.ai/free/en/papers/tang-2023-toolalpaca-generalized-tool-learning-language/). arXiv, abs/2306.05301.
31. Xiaoyan Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. AppAgent: Multimodal agents as smartphone users. arXiv, abs/2312.13771. [find](https://scholar.google.com/scholar?q=Xiaoyan+Zhang%2C+Zhao+Yang%2C+Jiaxuan+Liu%2C+Yucheng+Han%2C+Xin+Chen%2C+Zebiao+Huang%2C+Bin+Fu%2C+and+Gang+Yu%2E+2023%2E+AppAgent%3A+Multimodal+agents+as+smartphone+users%2E+arXiv%2C+abs%2F2312%2E13771%2E)
32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv, abs/2302.13971.
33. Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. 2024. MMInA: Benchmarking multihop multimodal internet agents. [find](https://scholar.google.com/scholar?q=Ziniu+Zhang%2C+Shulin+Tian%2C+Liangyu+Chen%2C+and+Ziwei+Liu%2E+2024%2E+MMInA%3A+Benchmarking+multihop+multimodal+internet+agents%2E)
34. Junyang Wang, Haiyang Xu, Jiabo Ye, Mingshi Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024a. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv, abs/2401.16158. — [arXiv:2401.16158](https://arxiv.org/abs/2401.16158) [find](https://scholar.google.com/scholar?q=Junyang+Wang%2C+Haiyang+Xu%2C+Jiabo+Ye%2C+Mingshi+Yan%2C+Weizhou+Shen%2C+Ji+Zhang%2C+Fei+Huang%2C+and+Jitao+Sang%2E+2024a%2E+Mobile%2Dagent%3A+Autonomous+multi%2Dmodal+mobile+device+agent+with+visual+perception%2E+arXiv%2C+abs%2F2)
35. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024b. [Executable code actions elicit better LLM agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/). arXiv, abs/2402.01030.
36. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024c. [MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback](https://md.research.elamur.ai/free/en/papers/wang-2023-mint-evaluating-llms-multi-turn/). In ICLR.
37. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. [STaR: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In NeurIPS.
38. Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. DocPrompting: Generating code by retrieving the docs. In ICLR. [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Uri+Alon%2C+Frank+F%2E+Xu%2C+Zhiruo+Wang%2C+Zhengbao+Jiang%2C+and+Graham+Neubig%2E+2022%2E+DocPrompting%3A+Generating+code+by+retrieving+the+docs%2E+In+ICLR%2E)
39. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. [WebArena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In ICLR.
40. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. [ToolQA: A dataset for LLM question answering with external tools](https://md.research.elamur.ai/free/en/papers/zhuang-2023-toolqa-dataset-llm-question-answering/). In NeurIPS: Datasets and Benchmarks Track.
41. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In ACL. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+ACL%2E)
42. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. [OS-World: Benchmarking multimodal agents for open-ended tasks i…](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/)
43. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. [On the tool manipulation capability of open-source large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-tool-manipulation-capability-open-source/). arXiv, abs/2305.16504.
44. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. [InterCode: Standardizing and benchmarking interactive coding with execution feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/). In NeurIPS: Datasets and Benchmarks Track. 16033 12

*Updated 24 October 2026*