---
title: "NaturalGAIA — paper by Zheng et al. (2026)"
description: "The authors also present LightManus-Jarvis. 49 references. GUI agent benchmarks tend to fall into two camps."
authors: ["Zihan Zheng", "Tianle Cui", "Taoran Wang", "Fengtao Wang", "Jiahui Pan", "Lewei He", "Qianglong Chen"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2207/"
doi: "10.18653/v1/2026.acl-long.2207"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zheng-2026-naturalgaia-verifiable-benchmark-hierarchical-framework/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zheng-2026-naturalgaia-verifiable-benchmark-hierarchical-framework/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/) / NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks

# NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks

**Authors:** Zihan Zheng, Tianle Cui, Taoran Wang, Fengtao Wang, [Jiahui Pan](https://md.research.elamur.ai/free/en/authors/jiahui-pan/), [Lewei He](https://md.research.elamur.ai/free/en/authors/lewei-he/) and [Qianglong Chen](https://md.research.elamur.ai/free/en/authors/qianglong-chen/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2207/) · [PDF](https://aclanthology.org/2026.acl-long.2207.pdf) · [DOI 10.18653/v1/2026.acl-long.2207](https://doi.org/10.18653/v1/2026.acl-long.2207) · [OpenAlex](https://openalex.org/W7166818502)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
GUI agent benchmarks tend to fall into two camps. Some are realistic but hard to verify because they depend on MLLM judges or manual checks. Others are verifiable but use simplified, context-free instructions. NaturalGAIA bridges this gap with 276 tasks whose logical backbone is a causal path over a knowledge graph such as Wikidata, so every atomic step has a deterministic answer. Steps are assigned to apps through entity attributes. Each path is then wrapped in natural language that reorders sub-intents, adds distracting noise and replaces explicit values with references to earlier results. Evaluation runs from difficulty-weighted task success down to per-step progress and error attribution. The authors also present LightManus-Jarvis. LightManus builds a task topology and passes context between steps, and Jarvis executes atomic actions using combined visual and structural perception. Driven by Claude-Sonnet-4.5, the system reaches 45.6% WPSR, against 21.1% for Mobile-Agent-e, while using roughly a quarter of the tokens and time. This gives a reproducible way to measure the reasoning-execution gap in long-horizon GUI tasks.

## Key points
- NaturalGAIA has 276 tasks built in three stages: path generation, tool grounding and narrative injection. Its notion of naturalness spans five dimensions: chaining steps over a long horizon, grounding actions across different domains, non-linear ordering of the user's thinking, redundant or distracting information, and dependence on earlier context.
- The metrics are WPSR, where success is weighted by path length and number of apps, plus MATCR and p-ATSR, which measure partial progress and give more weight to later steps.
- WPSR results: LightManus-Jarvis scores 45.6%, ahead of Mobile-Agent-e (21.1%) and PC-Agent (13.1%).
- On Level-3 tasks, Jarvis needs 112,600 tokens versus 427,980 for Mobile-Agent-e, and 505.8 s versus 2015.4 s, about 74–75% less of each.
- Error analysis covers five types: KD, PRE, OE, PE and SCE. Operational errors dominate at 19–30%. Replacing the Mobile-Agent-e executor with Jarvis lowered PE from 7.1% to 3.3% for Gemini-2.5-Pro. Limitations include excluding irreversible or dynamic scenarios such as e-commerce and relying on human validation of tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🖱️ Human-Computer Interaction](https://md.research.elamur.ai/free/en/shelves/computer-science/human-computer-interaction/) · 160 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Zihan Zheng, Tianle Cui, Taoran Wang, Fengtao Wang, Jiahui Pan, Lewei He and Qianglong Chen. 2026. NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zheng2026naturalgaia,
  title = {{NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks}},
  author = {Zihan Zheng, Tianle Cui, Taoran Wang, Fengtao Wang, Jiahui Pan, Lewei He and Qianglong Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2207},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2207}
}
```

- [← previous Natural-Language Policies to Executable Decisions: An Interpretable Large Language Model Framework](https://md.research.elamur.ai/free/en/papers/zhang-2026-natural-language-policies-executable-decisions/)
- [next → Navigating the Infinite Dynamic Web Space: Effective In-Context Exploration via Cognitive Multi-Agent Collaboration](https://md.research.elamur.ai/free/en/papers/mo-2026-navigating-infinite-dynamic-web-space/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/43/)

## References

49 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mathieu Andreux, Märt Bakler, Yanael Barbier, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Aleix Cam-bray, Pierre-Louis Cedoz, Antoine Chassang, Gautier Cloix, Ethan Connelly, Alexandra Constanti-nou, Ramzi De Coster, Hubert de la Jonquiere, Aurélien De… — [arXiv:2510.19949](https://arxiv.org/abs/2510.19949) [find](https://scholar.google.com/scholar?q=Mathieu+Andreux%2C+M%C3%A4rt+Bakler%2C+Yanael+Barbier%2C+Hamza+Benchekroun%2C+Emilien+Bir%C3%A9%2C+Antoine+Bonnet%2C+Riaz+Bordie%2C+Nathan+Bout%2C+Matthias+Brunel%2C+Aleix+Cam%2Dbray%2C+Pierre%2DLouis+Cedoz%2C+Antoine+Chassang%2C+Gautier+)
2. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. 2025. Agentharm: A benchmark for measuring harmfulness of llm agents. Preprint, arXiv:2410… — [arXiv:2410.09024](https://arxiv.org/abs/2410.09024) [find](https://scholar.google.com/scholar?q=Maksym+Andriushchenko%2C+Alexandra+Souly%2C+Mateusz+Dziemian%2C+Derek+Duenas%2C+Maxwell+Lin%2C+Justin+Wang%2C+Dan+Hendrycks%2C+Andy+Zou%2C+Zico+Kolter%2C+Matt+Fredrikson%2C+Eric+Winsor%2C+Jerome+Wynne%2C+Yarin+Gal%2C+and+Xande)
3. Zhicheng Cai, Xinyuan Guo, Yu Pei, Jiangtao Feng, Jinsong Su, Jiangjie Chen, Ya-Qin Zhang, Wei-Ying Ma, Mingxuan Wang, and Hao Zhou. 2025. Flex: Continuous agent evolution via forward learning from experience. Preprint, arXiv:2511.06449. — [arXiv:2511.06449](https://arxiv.org/abs/2511.06449) [find](https://scholar.google.com/scholar?q=Zhicheng+Cai%2C+Xinyuan+Guo%2C+Yu+Pei%2C+Jiangtao+Feng%2C+Jinsong+Su%2C+Jiangjie+Chen%2C+Ya%2DQin+Zhang%2C+Wei%2DYing+Ma%2C+Mingxuan+Wang%2C+and+Hao+Zhou%2E+2025%2E+Flex%3A+Continuous+agent+evolution+via+forward+learning+from+ex)
4. Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guangyi Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. 2025. A3: Android agent arena for mobile gui agents. Preprint, arXiv:2501.01149. — [arXiv:2501.01149](https://arxiv.org/abs/2501.01149) [find](https://scholar.google.com/scholar?q=Yuxiang+Chai%2C+Hanhao+Li%2C+Jiayu+Zhang%2C+Liang+Liu%2C+Guangyi+Liu%2C+Guozhi+Wang%2C+Shuai+Ren%2C+Siyuan+Huang%2C+and+Hongsheng+Li%2E+2025%2E+A3%3A+Android+agent+arena+for+mobile+gui+agents%2E+Preprint%2C+arXiv%3A2501%2E01149%2E+%E2%80%94)
5. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. [SeeClick: Harnessing GUI grounding for advanced visual GUI agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313– 9332,…
6. Wonje Choi, Jinwoo Park, Sanghyun Ahn, Daehee Lee, and Honguk Woo. 2025. Nesyc: A neuro-symbolic continual learner for complex embodied tasks in open domains. Preprint, arXiv:2503.00870. — [arXiv:2503.00870](https://arxiv.org/abs/2503.00870) [find](https://scholar.google.com/scholar?q=Wonje+Choi%2C+Jinwoo+Park%2C+Sanghyun+Ahn%2C+Daehee+Lee%2C+and+Honguk+Woo%2E+2025%2E+Nesyc%3A+A+neuro%2Dsymbolic+continual+learner+for+complex+embodied+tasks+in+open+domains%2E+Preprint%2C+arXiv%3A2503%2E00870%2E+%E2%80%94+%5BarXiv%3A2503)
7. Yutong Dai, Krithika Ramakrishnan, Jing Gu, Matthew Fernandez, Yanqi Luo, Viraj Prabhu, Zhenyu Hu, Silvio Savarese, Caiming Xiong, Zeyuan Chen, and Ran Xu. 2025. Scuba: Salesforce computer use benchmark. Preprint, arXiv:2509.26506. — [arXiv:2509.26506](https://arxiv.org/abs/2509.26506) [find](https://scholar.google.com/scholar?q=Yutong+Dai%2C+Krithika+Ramakrishnan%2C+Jing+Gu%2C+Matthew+Fernandez%2C+Yanqi+Luo%2C+Viraj+Prabhu%2C+Zhenyu+Hu%2C+Silvio+Savarese%2C+Caiming+Xiong%2C+Zeyuan+Chen%2C+and+Ran+Xu%2E+2025%2E+Scuba%3A+Salesforce+computer+use+benchma)
8. DeepSeek-AI and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
9. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). Preprint, arXiv:2306.06070.
10. Lingzhong Dong, Ziqi Zhou, Shuaibo Yang, Haiyue Sheng, Pengzhou Cheng, Zongru Wu, Zheng Wu, Gongshen Liu, and Zhuosheng Zhang. 2025. Say one thing, do another? diagnosing reasoning-execution gaps in vlm-powered mobile-use agents. Preprint, arXiv:2510.02204. — [arXiv:2510.02204](https://arxiv.org/abs/2510.02204) [find](https://scholar.google.com/scholar?q=Lingzhong+Dong%2C+Ziqi+Zhou%2C+Shuaibo+Yang%2C+Haiyue+Sheng%2C+Pengzhou+Cheng%2C+Zongru+Wu%2C+Zheng+Wu%2C+Gongshen+Liu%2C+and+Zhuosheng+Zhang%2E+2025%2E+Say+one+thing%2C+do+another%3F+diagnosing+reasoning%2Dexecution+gaps+in+v)
11. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. 2024. Workarena: How capable are web agents at solving common knowledge work tasks? Preprint, arXiv:2403… — [arXiv:2403.07718](https://arxiv.org/abs/2403.07718) [find](https://scholar.google.com/scholar?q=Alexandre+Drouin%2C+Maxime+Gasse%2C+Massimo+Caccia%2C+Issam+H%2E+Laradji%2C+Manuel+Del+Verme%2C+Tom+Marty%2C+L%C3%A9o+Boisvert%2C+Megh+Thakkar%2C+Quentin+Cappart%2C+David+Vazquez%2C+Nicolas+Chapados%2C+and+Alexandre+Lacoste%2E+2024)
12. Qihang Fang, Chengcheng Tang, Shugao Ma, and Yanchao Yang. 2024. Bid: Boundary-interior decoding for unsupervised temporal action localization pre-trainin. Preprint, arXiv:2403.07354. — [arXiv:2403.07354](https://arxiv.org/abs/2403.07354) [find](https://scholar.google.com/scholar?q=Qihang+Fang%2C+Chengcheng+Tang%2C+Shugao+Ma%2C+and+Yanchao+Yang%2E+2024%2E+Bid%3A+Boundary%2Dinterior+decoding+for+unsupervised+temporal+action+localization+pre%2Dtrainin%2E+Preprint%2C+arXiv%3A2403%2E07354%2E+%E2%80%94+%5BarXiv%3A2403%2E07)
13. Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden, and Asaf Yehudai. 2025. [JuStRank: Benchmarking LLM judges for system ranking](https://md.research.elamur.ai/free/en/papers/gera-2025-justrank-benchmarking-llm-judges-system/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 682–712, Vienna, Austria. Associ…
14. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024a. [WebVoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
15. Yanheng He, Jiahe Jin, and Pengfei Liu. 2025. Efficient agent training for computer use. Preprint, arXiv:2505.13909. — [arXiv:2505.13909](https://arxiv.org/abs/2505.13909) [find](https://scholar.google.com/scholar?q=Yanheng+He%2C+Jiahe+Jin%2C+and+Pengfei+Liu%2E+2025%2E+Efficient+agent+training+for+computer+use%2E+Preprint%2C+arXiv%3A2505%2E13909%2E+%E2%80%94+%5BarXiv%3A2505%2E13909%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E13909%29)
16. Yanheng He, Jiahe Jin, Shijie Xia, Jiadi Su, Runze Fan, Haoyang Zou, Xiangkun Hu, and Pengfei Liu. 2024b. Pc agent: While you sleep, ai works - a cognitive journey into digital world. arXiv preprint arXiv:2412.17589. — [arXiv:2412.17589](https://arxiv.org/abs/2412.17589) [find](https://scholar.google.com/scholar?q=Yanheng+He%2C+Jiahe+Jin%2C+Shijie+Xia%2C+Jiadi+Su%2C+Runze+Fan%2C+Haoyang+Zou%2C+Xiangkun+Hu%2C+and+Pengfei+Liu%2E+2024b%2E+Pc+agent%3A+While+you+sleep%2C+ai+works+%2D+a+cognitive+journey+into+digital+world%2E+arXiv+preprint+a)
17. Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. [Metagpt: Meta programming for a multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). Preprint, a…
18. Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai, Caiming Xiong, and Chien-Sheng Wu. 2025. Gui-kv: Efficient gui agents via kv cache with spatio-temporal awareness. Preprint, arXiv:2510.00536. — [arXiv:2510.00536](https://arxiv.org/abs/2510.00536) [find](https://scholar.google.com/scholar?q=Kung%2DHsiang+Huang%2C+Haoyi+Qiu%2C+Yutong+Dai%2C+Caiming+Xiong%2C+and+Chien%2DSheng+Wu%2E+2025%2E+Gui%2Dkv%3A+Efficient+gui+agents+via+kv+cache+with+spatio%2Dtemporal+awareness%2E+Preprint%2C+arXiv%3A2510%2E00536%2E+%E2%80%94+%5BarXiv%3A2510%2E0)
19. NVJK Kartik, Garvit Sapra, Rishav Hada, and Nikhil Pareek. 2025. Agentcompass: Towards reliable evaluation of agentic workflows in production. Preprint, arXiv:2509.14647. — [arXiv:2509.14647](https://arxiv.org/abs/2509.14647) [find](https://scholar.google.com/scholar?q=NVJK+Kartik%2C+Garvit+Sapra%2C+Rishav+Hada%2C+and+Nikhil+Pareek%2E+2025%2E+Agentcompass%3A+Towards+reliable+evaluation+of+agentic+workflows+in+production%2E+Preprint%2C+arXiv%3A2509%2E14647%2E+%E2%80%94+%5BarXiv%3A2509%2E14647%5D%28https%3A%2F%2F)
20. Ziyue Li, Yuan Chang, Gaihong Yu, and Xiaoqiu Le. 2025. Hiplan: Hierarchical planning for llm-based agents with adaptive global-local guidance. Preprint, arXiv:2508.19076. — [arXiv:2508.19076](https://arxiv.org/abs/2508.19076) [find](https://scholar.google.com/scholar?q=Ziyue+Li%2C+Yuan+Chang%2C+Gaihong+Yu%2C+and+Xiaoqiu+Le%2E+2025%2E+Hiplan%3A+Hierarchical+planning+for+llm%2Dbased+agents+with+adaptive+global%2Dlocal+guidance%2E+Preprint%2C+arXiv%3A2508%2E19076%2E+%E2%80%94+%5BarXiv%3A2508%2E19076%5D%28https%3A%2F)
21. Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, Ke Li, and Xing Sun. 2025. Cuareward-bench: A benchmark for evaluating reward models on computer-using agent. Preprint, arXiv:2510.18596. — [arXiv:2510.18596](https://arxiv.org/abs/2510.18596) [find](https://scholar.google.com/scholar?q=Haojia+Lin%2C+Xiaoyu+Tan%2C+Yulei+Qin%2C+Zihan+Xu%2C+Yuchen+Shi%2C+Zongyi+Li%2C+Gang+Li%2C+Shaofei+Cai%2C+Siqi+Cai%2C+Chaoyou+Fu%2C+Ke+Li%2C+and+Xing+Sun%2E+2025%2E+Cuareward%2Dbench%3A+A+benchmark+for+evaluating+reward+models+on+)
22. Jiyuan Liu, Jielin Song, Yunhe Pang, Zhiyu Shen, and Yanghui Rao. 2025. CARE: A disagreement detection framework with concept alignment and reasoning enhancement. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 13275–13290, Suzhou, China. Association… [find](https://scholar.google.com/scholar?q=Jiyuan+Liu%2C+Jielin+Song%2C+Yunhe+Pang%2C+Zhiyu+Shen%2C+and+Yanghui+Rao%2E+2025%2E+CARE%3A+A+disagreement+detection+framework+with+concept+alignment+and+reasoning+enhancement%2E+In+Proceedings+of+the+2025+Conference)
23. Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, and Siva Reddy. 2025. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. Preprint, arXiv:2504.08942. — [arXiv:2504.08942](https://arxiv.org/abs/2504.08942) [find](https://scholar.google.com/scholar?q=Xing+Han+L%C3%B9%2C+Amirhossein+Kazemnejad%2C+Nicholas+Meade%2C+Arkil+Patel%2C+Dongchan+Shin%2C+Alejandra+Zambrano%2C+Karolina+Stan%CC%81czak%2C+Peter+Shaw%2C+Christopher+J%2E+Pal%2C+and+Siva+Reddy%2E+2025%2E+Agentrewardbench%3A+Evaluat)
24. Yansheng Mao, Yufei Xu, Jiaqi Li, Fanxu Meng, Haotong Yang, Zilong Zheng, Xiyuan Wang, and Muhan Zhang. 2025. Lift: Improving long context understanding of large language models through long input fine-tuning. Preprint, arXiv:2502.14644. — [arXiv:2502.14644](https://arxiv.org/abs/2502.14644) [find](https://scholar.google.com/scholar?q=Yansheng+Mao%2C+Yufei+Xu%2C+Jiaqi+Li%2C+Fanxu+Meng%2C+Haotong+Yang%2C+Zilong+Zheng%2C+Xiyuan+Wang%2C+and+Muhan+Zhang%2E+2025%2E+Lift%3A+Improving+long+context+understanding+of+large+language+models+through+long+input+fin)
25. Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and benchmarking of llm agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, page 6129–6139, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Mahmoud+Mohammadi%2C+Yipeng+Li%2C+Jane+Lo%2C+and+Wendy+Yip%2E+2025%2E+Evaluation+and+benchmarking+of+llm+agents%3A+A+survey%2E+In+Proceedings+of+the+31st+ACM+SIGKDD+Conference+on+Knowledge+Discovery+and+Data+Mining)
26. Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024. Screenagent: A vision language model-driven computer control agent. [find](https://scholar.google.com/scholar?q=Runliang+Niu%2C+Jindong+Li%2C+Shiqi+Wang%2C+Yali+Fu%2C+Xiyu+Hu%2C+Xueyuan+Leng%2C+He+Kong%2C+Yi+Chang%2C+and+Qi+Wang%2E+2024%2E+Screenagent%3A+A+vision+language+model%2Ddriven+computer+control+agent%2E)
27. OpenAI and 1 others. 2024. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). Preprint, arXiv:2412.16720.
28. Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi, Jitendra Malik, and Francois Hogan. 2025. Spider: Scalable physics-informed dexterous retargeting. Preprint, arXiv:2511.09484. — [arXiv:2511.09484](https://arxiv.org/abs/2511.09484) [find](https://scholar.google.com/scholar?q=Chaoyi+Pan%2C+Changhao+Wang%2C+Haozhi+Qi%2C+Zixi+Liu%2C+Homanga+Bharadhwaj%2C+Akash+Sharma%2C+Tingfan+Wu%2C+Guanya+Shi%2C+Jitendra+Malik%2C+and+Francois+Hogan%2E+2025%2E+Spider%3A+Scalable+physics%2Dinformed+dexterous+retarget)
29. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025. [Ui-tars: Pioneering automated gui interaction with…](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/)
30. Qwen and 1 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen+and+1+others%2E+2025%2E+Qwen2%2E5+technical+report%2E+Preprint%2C+arXiv%3A2412%2E15115%2E)
31. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. 2024. [Androidworld: A dynamic benchmarking environment f…](https://md.research.elamur.ai/free/en/papers/rawles-2024-androidworld-dynamic-benchmarking-environment-autonomous/)
32. Martin Riddell, Ansong Ni, and Arman Cohan. 2024. Quantifying contamination in evaluating code generation capabilities of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14116–14137, Bangkok, Thailand. Associa… [find](https://scholar.google.com/scholar?q=Martin+Riddell%2C+Ansong+Ni%2C+and+Arman+Cohan%2E+2024%2E+Quantifying+contamination+in+evaluating+code+generation+capabilities+of+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+)
33. Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. 2025. Agentsquare: Automatic llm agent search in modular design space. Preprint, arXiv:2410.06153. — [arXiv:2410.06153](https://arxiv.org/abs/2410.06153) [find](https://scholar.google.com/scholar?q=Yu+Shang%2C+Yu+Li%2C+Keyu+Zhao%2C+Likai+Ma%2C+Jiahe+Liu%2C+Fengli+Xu%2C+and+Yong+Li%2E+2025%2E+Agentsquare%3A+Automatic+llm+agent+search+in+modular+design+space%2E+Preprint%2C+arXiv%3A2410%2E06153%2E+%E2%80%94+%5BarXiv%3A2410%2E06153%5D%28https%3A%2F)
34. Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Xiangru Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, and 2 others. 2025. Scienceboard: Evaluating multimodal aut… [find](https://scholar.google.com/scholar?q=Qiushi+Sun%2C+Zhoumianze+Liu%2C+Chang+Ma%2C+Zichen+Ding%2C+Fangzhi+Xu%2C+Zhangyue+Yin%2C+Haiteng+Zhao%2C+Zhenyu+Wu%2C+Kanzhi+Cheng%2C+Zhaoyang+Liu%2C+Jianing+Wang%2C+Qintong+Li%2C+Xiangru+Tang%2C+Tianbao+Xie%2C+Xiachong+Feng%2C+Xi)
35. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. [Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/). Preprint, arXiv:2206.10498.
36. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, and 93 others. 2025a. [Ui-tars-2 technical report: Advancing gui ag…](https://md.research.elamur.ai/free/en/papers/wang-2025-ui-tars-2-technical/)
37. Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. 2025b. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733. — [arXiv:2501.11733](https://arxiv.org/abs/2501.11733) [find](https://scholar.google.com/scholar?q=Zhenhailong+Wang%2C+Haiyang+Xu%2C+Junyang+Wang%2C+Xi+Zhang%2C+Ming+Yan%2C+Ji+Zhang%2C+Fei+Huang%2C+and+Heng+Ji%2E+2025b%2E+Mobile%2Dagent%2De%3A+Self%2Devolving+mobile+assistant+for+complex+tasks%2E+arXiv+preprint+arXiv%3A2501%2E117)
38. Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. 2025. [PlanGenLLMs: A modern survey of LLM planning capabilities](https://md.research.elamur.ai/free/en/papers/wei-2025-plangenllms-modern-survey-llm-planning/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19497–19521, Vienna, Austria. Association f…
39. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. [Osworld: Benchmarking multimodal agents for open-ended tasks in…](https://md.research.elamur.ai/free/en/papers/xie-2024-osworld-benchmarking-multimodal-agents-open/)
40. Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. 2024. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. [find](https://scholar.google.com/scholar?q=Yiheng+Xu%2C+Dunjie+Lu%2C+Zhennan+Shen%2C+Junli+Wang%2C+Zekun+Wang%2C+Yuchen+Mao%2C+Caiming+Xiong%2C+and+Tao+Yu%2E+2024%2E+Agenttrek%3A+Agent+trajectory+synthesis+via+guiding+replay+with+web+tutorials%2E)
41. Zhen Yang, Zi-Yi Dou, Di Feng, Forrest Huang, Anh Nguyen, Keen You, Omar Attia, Yuhao Yang, Michael Feng, Haotian Zhang, Ram Ramrakhya, Chao Jia, Jeffrey Nichols, Alexander Toshev, Yinfei Yang, and Zhe Gan. 2025. Ferret-ui lite: Lessons from building small on-device gui agents. Preprint, arXiv:2509.… — [arXiv:2509.26539](https://arxiv.org/abs/2509.26539) [find](https://scholar.google.com/scholar?q=Zhen+Yang%2C+Zi%2DYi+Dou%2C+Di+Feng%2C+Forrest+Huang%2C+Anh+Nguyen%2C+Keen+You%2C+Omar+Attia%2C+Yuhao+Yang%2C+Michael+Feng%2C+Haotian+Zhang%2C+Ram+Ramrakhya%2C+Chao+Jia%2C+Jeffrey+Nichols%2C+Alexander+Toshev%2C+Yinfei+Yang%2C+and+Zh)
42. Long-Horizon Composition: To challenge the agent’s context maintenance capabilities, each main task must be composed of a sequence of atomic interactions (typically N ≥ 6). These atomic tasks are explicitly delimited in the logical skeleton, preventing ambiguous task boundaries. [find](https://scholar.google.com/scholar?q=Long%2DHorizon+Composition%3A+To+challenge+the+agent%E2%80%99s+context+maintenance+capabilities%2C+each+main+task+must+be+composed+of+a+sequence+of+atomic+interactions+%28typically+N+%E2%89%A5+6%29%2E+These+atomic+tasks+are+expl)
43. Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, and Tianmin Shu. 2025. Realwebassist: A benchmark for long-horizon web assistance with real-world users. Preprint, arXiv:2504.10445. — [arXiv:2504.10445](https://arxiv.org/abs/2504.10445) [find](https://scholar.google.com/scholar?q=Suyu+Ye%2C+Haojun+Shi%2C+Darren+Shih%2C+Hyokun+Yun%2C+Tanya+Roosta%2C+and+Tianmin+Shu%2E+2025%2E+Realwebassist%3A+A+benchmark+for+long%2Dhorizon+web+assistance+with+real%2Dworld+users%2E+Preprint%2C+arXiv%3A2504%2E10445%2E+%E2%80%94+%5BarXi)
44. Zecheng Yin, Hao Zhao, and Zhen Li. 2025. Hypernav: Hybrid perception for object-oriented navigation in unknown environment. Preprint, arXiv:2510.22917. — [arXiv:2510.22917](https://arxiv.org/abs/2510.22917) [find](https://scholar.google.com/scholar?q=Zecheng+Yin%2C+Hao+Zhao%2C+and+Zhen+Li%2E+2025%2E+Hypernav%3A+Hybrid+perception+for+object%2Doriented+navigation+in+unknown+environment%2E+Preprint%2C+arXiv%3A2510%2E22917%2E+%E2%80%94+%5BarXiv%3A2510%2E22917%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2510)
45. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. [Appagent: Multimodal agents as smartphone users](https://md.research.elamur.ai/free/en/papers/zhang-2023-appagent-multimodal-agents-smartphone-users/). Preprint, arXiv:2312.13771.
46. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. 2025a. Agentic context engineering: Evolving contexts for self-improving language models. Preprint, arXiv:2510.0461… [find](https://scholar.google.com/scholar?q=Qizheng+Zhang%2C+Changran+Hu%2C+Shubhangi+Upasani%2C+Boyuan+Ma%2C+Fenglu+Hong%2C+Vamsidhar+Kamanuru%2C+Jay+Rainton%2C+Chen+Wu%2C+Mengmeng+Ji%2C+Hanchen+Li%2C+Urmish+Thakker%2C+James+Zou%2C+and+Kunle+Olukotun%2E+2025a%2E+Agentic+)
47. Multi-App Collaboration: The task execution must necessitate context switching between distinct application environments. We enforce a minimum diversity constraint (e.g., interacting with at least three distinct applications), ensuring the agent is evaluated on cross-domain grounding rather than sin… [find](https://scholar.google.com/scholar?q=Multi%2DApp+Collaboration%3A+The+task+execution+must+necessitate+context+switching+between+distinct+application+environments%2E+We+enforce+a+minimum+diversity+constraint+%28e%2Eg%2E%2C+interacting+with+at+least+thr)
48. Explicit Textual Output: Every atomic task must yield a verifiable result presentable in text format. This enables automated evaluation of intermediate steps. • Uniqueness and Objectivity: The output space of each atomic task must be constrained to a single, unambiguous ground truth (Deterministic C… [find](https://scholar.google.com/scholar?q=Explicit+Textual+Output%3A+Every+atomic+task+must+yield+a+verifiable+result+presentable+in+text+format%2E+This+enables+automated+evaluation+of+intermediate+steps%2E+%E2%80%A2+Uniqueness+and+Objectivity%3A+The+output+)
49. Zihan Zheng, Tianle Cui, Chuwen Xie, Jiahui Pan, Qianglong Chen, and Lewei He. 2025. PlanningArena: A modular benchmark for multidimensional evaluation of planning and tool learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… [find](https://scholar.google.com/scholar?q=Zihan+Zheng%2C+Tianle+Cui%2C+Chuwen+Xie%2C+Jiahui+Pan%2C+Qianglong+Chen%2C+and+Lewei+He%2E+2025%2E+PlanningArena%3A+A+modular+benchmark+for+multidimensional+evaluation+of+planning+and+tool+learning%2E+In+Proceedings+of)

*Updated 2 November 2026*