---
title: "Measure Twice, Click Once — paper by Wang et al. (2026)"
description: "The authors propose COPC, a Propose-then-Critic framework. 46 references. Among them: 0."
authors: ["Wenkai Wang", "Xiyun Li", "Hongcan Guo", "Wenhao Yu", "Tianqing Fang", "Haitao Mi", "Dong Yu", "Shengyu Zhang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.960/"
doi: "10.18653/v1/2026.acl-long.960"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-measure-twice-click-once-co/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-measure-twice-click-once-co/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/) / Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding

# Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding

**Authors:** [Wenkai Wang](https://md.research.elamur.ai/free/en/authors/wenkai-wang/), Xiyun Li, Hongcan Guo, [Wenhao Yu](https://md.research.elamur.ai/free/en/authors/wenhao-yu/), [Tianqing Fang](https://md.research.elamur.ai/free/en/authors/tianqing-fang/), [Haitao Mi](https://md.research.elamur.ai/free/en/authors/haitao-mi/), [Dong Yu](https://md.research.elamur.ai/free/en/authors/dong-yu/) and [Shengyu Zhang](https://md.research.elamur.ai/free/en/authors/shengyu-zhang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.960/) · [PDF](https://aclanthology.org/2026.acl-long.960.pdf) · [DOI 10.18653/v1/2026.acl-long.960](https://doi.org/10.18653/v1/2026.acl-long.960) · [OpenAlex](https://openalex.org/W7166841832)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
GUI grounding means turning a natural-language instruction into exact click coordinates on a screenshot. MLLMs find it hard on high-resolution, crowded interfaces: they often understand the request but miss the precise pixel location. Drawing several samples raises Pass@k. However, clustering-style self-consistency (mean, median, medoid) helps little, because the predicted points scatter widely. Off-the-shelf models are also weak at judging plausible candidates.

The authors propose COPC, a Propose-then-Critic framework. A single MLLM first outputs several candidate points in one pass. These points are drawn on the screenshot with ID markers, and the same model then ranks them visually. Both roles are trained jointly with GRPO-based reinforcement learning using separate rewards:
- accuracy and spatial coverage for the proposer;
- top-1 hit and NDCG-style ranking for the critic.

Maturity estimates weight these rewards so that training moves from basic localization toward broader exploration. Across six benchmarks, the method gives up to 17.2% relative grounding improvement. It also shrinks the gap between oracle and selected accuracy, which shows that learned visual self-critique outperforms geometric aggregation.

## Key points
- Evaluation covers ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, MMBench-GUI, I2E-bench and UI-Vision (Basic/Func/Spatial). The method is compared with Direct SFT, Two-Stage SFT, DPO and GRPO, and it shows the smallest Oracle@5 vs. Top-1 gap.
- On ScreenSpot-Pro, Top-1 reaches 56.4 (Oracle@5 59.8), compared with 53.4 for plain GRPO and 29.3 for the base model.
- The ablation on Qwen3-VL-4B gives 70.0 Oracle@5 and 67.3 Top-1 for the full model. Replacing maturity-based weighting with static weights drops these to 62.6 and 57.5, the largest loss. Removing the coverage or ranking rewards also hurts.
- Candidates come from one inference pass, so the method uses fewer tokens than spatial baselines that aggregate 8 independent runs, while still beating them. Gains carry over to Qwen2.5-VL 3B/7B, and results are robust to the momentum coefficient α.
- Stated limitations: latency is higher than single-step regression, and rendered markers may partly hide tiny elements in very dense layouts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Wenkai Wang, Xiyun Li, Hongcan Guo, Wenhao Yu, Tianqing Fang, Haitao Mi, Dong Yu and Shengyu Zhang. 2026. Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026measure,
  title = {{Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding}},
  author = {Wenkai Wang, Xiyun Li, Hongcan Guo, Wenhao Yu, Tianqing Fang, Haitao Mi, Dong Yu and Shengyu Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.960},
  url = {https://doi.org/10.18653/v1/2026.acl-long.960}
}
```

- [← previous Meaning-Making Process and Error Dynamics in ChatGPT-Mediated Translation](https://md.research.elamur.ai/free/en/papers/mihalache-2026-meaning-making-process-error-dynamics/)
- [next → Measuring and Mitigating Shortcut Reliance in Language Models with Probe-Based Representation Entanglement](https://md.research.elamur.ai/free/en/papers/singh-2026-measuring-mitigating-shortcut-reliance-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/)

## References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Developing computer use. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) developing-computer-use. Accessed: 2026-01-01. [↗](https://www.anthropic.com/news/)
2. Anthropic. 2025. Claude 3.7 sonnet system card. Technical report, Anthropic. Accessed: 2025-08-02. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+Claude+3%2E7+sonnet+system+card%2E+Technical+report%2C+Anthropic%2E+Accessed%3A+2025%2D08%2D02%2E)
3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. [Qwen3-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). arXiv prepr…
4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. [Qwen2.5-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/). arXiv preprint arXiv:2…
5. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, and 1 others. 2025. Guicourse: From general vision language model to versatile gui agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (… [find](https://scholar.google.com/scholar?q=Wentong+Chen%2C+Junbo+Cui%2C+Jinyi+Hu%2C+Yujia+Qin%2C+Junjie+Fang%2C+Yue+Zhao%2C+Chongyi+Wang%2C+Jun+Liu%2C+Guirong+Chen%2C+Yupeng+Huo%2C+and+1+others%2E+2025%2E+Guicourse%3A+From+general+vision+language+model+to+versatile+gui)
6. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. [Seeclick: Harnessing gui grounding for advanced visual gui agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332.
7. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. 2025. [Sft memorizes, rl generalizes: A comparative study of foundation model post-training](https://md.research.elamur.ai/free/en/papers/chu-2025-sft-memorizes-rl-generalizes-comparative/). arXiv preprint arXiv:2501.17161.
8. Yong Du, Yuchen Yan, Fei Tang, Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, and Yongliang Shen. 2025. Test-time reinforcement learning for gui grounding via region consistency. arXiv preprint arXiv:2508.05615. — [arXiv:2508.05615](https://arxiv.org/abs/2508.05615) [find](https://scholar.google.com/scholar?q=Yong+Du%2C+Yuchen+Yan%2C+Fei+Tang%2C+Zhengxi+Lu%2C+Chang+Zong%2C+Weiming+Lu%2C+Shengpei+Jiang%2C+and+Yongliang+Shen%2E+2025%2E+Test%2Dtime+reinforcement+learning+for+gui+grounding+via+region+consistency%2E+arXiv+preprint+a)
9. Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. 2025. [Webevolver: Enhancing web agent self-improvement with co-evolving world model](https://md.research.elamur.ai/free/en/papers/fang-2025-webevolver-enhancing-web-agent-self/). In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8970– 8986.
10. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243. [find](https://scholar.google.com/scholar?q=Boyu+Gou%2C+Ruohan+Wang%2C+Boyuan+Zheng%2C+Yanan+Xie%2C+Cheng+Chang%2C+Yiheng+Shu%2C+Huan+Sun%2C+and+Yu+Su%2E+2024%2E+Navigating+the+digital+world+as+humans+do%3A+Universal+visual+grounding+for+gui+agents%2E+arXiv+preprint)
11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025a. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
12. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, and 1 others. 2025b. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062. — [Seed1.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/guo-2025-seed1-5-vl-technical/)
13. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and 1 others. 2024. [Cogagent: A visual language model for gui agents](https://md.research.elamur.ai/free/en/papers/hong-2023-cogagent-visual-language-model/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290.
14. Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. 2025. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749. [find](https://scholar.google.com/scholar?q=Wenxuan+Huang%2C+Bohan+Jia%2C+Zijie+Zhai%2C+Shaosheng+Cao%2C+Zheyu+Ye%2C+Fei+Zhao%2C+Zhe+Xu%2C+Yao+Hu%2C+and+Shaohui+Lin%2E+2025%2E+Vision%2Dr1%3A+Incentivizing+reasoning+capability+in+multimodal+large+language+models%2E+arXiv)
15. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
16. Weitai Kang, Bin Lei, Gaowen Liu, Caiwen Ding, and Yan Yan. 2025. Guirlvg: Incentivize gui visual grounding via empirical exploration on reinforcement learning. arXiv preprint arXiv:2508.04389. — [arXiv:2508.04389](https://arxiv.org/abs/2508.04389) [find](https://scholar.google.com/scholar?q=Weitai+Kang%2C+Bin+Lei%2C+Gaowen+Liu%2C+Caiwen+Ding%2C+and+Yan+Yan%2E+2025%2E+Guirlvg%3A+Incentivize+gui+visual+grounding+via+empirical+exploration+on+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2508%2E04389%2E+%E2%80%94+%5BarX)
17. Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. 2024. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pages 161– 178. Springer. [find](https://scholar.google.com/scholar?q=Raghav+Kapoor%2C+Yash+Parag+Butala%2C+Melisa+Russak%2C+Jing+Yu+Koh%2C+Kiran+Kamble%2C+Waseem+AlShikh%2C+and+Ruslan+Salakhutdinov%2E+2024%2E+Omniact%3A+A+dataset+and+benchmark+for+enabling+multimodal+generalist+autonomo)
18. Bin Lei, Nuo Xu, Ali Payani, Mingyi Hong, Chunhua Liao, Yu Cao, and Caiwen Ding. 2025. \\textsc {GUI-Spotlight}: Adaptive iterative focus refinement for enhanced gui visual grounding. arXiv preprint arXiv:2510.04039. — [arXiv:2510.04039](https://arxiv.org/abs/2510.04039) [find](https://scholar.google.com/scholar?q=Bin+Lei%2C+Nuo+Xu%2C+Ali+Payani%2C+Mingyi+Hong%2C+Chunhua+Liao%2C+Yu+Cao%2C+and+Caiwen+Ding%2E+2025%2E+%5C%5Ctextsc+%7BGUI%2DSpotlight%7D%3A+Adaptive+iterative+focus+refinement+for+enhanced+gui+visual+grounding%2E+arXiv+preprint+a)
19. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786. [find](https://scholar.google.com/scholar?q=Kaixin+Li%2C+Ziyang+Meng%2C+Hongzhan+Lin%2C+Ziyang+Luo%2C+Yuchen+Tian%2C+Jing+Ma%2C+Zhiyong+Huang%2C+and+Tat%2DSeng+Chua%2E+2025%2E+Screenspot%2Dpro%3A+Gui+grounding+for+professional+high%2Dresolution+computer+use%2E+In+Proceedi)
20. Mukai Li, Qingcheng Zeng, Tianqing Fang, Zhenwen Liang, Linfeng Song, Qi Liu, Haitao Mi, and Dong Yu. 2026. Verified critical step optimization for llm agents. arXiv preprint arXiv:2602.03412. [find](https://scholar.google.com/scholar?q=Mukai+Li%2C+Qingcheng+Zeng%2C+Tianqing+Fang%2C+Zhenwen+Liang%2C+Linfeng+Song%2C+Qi+Liu%2C+Haitao+Mi%2C+and+Dong+Yu%2E+2026%2E+Verified+critical+step+optimization+for+llm+agents%2E+arXiv+preprint+arXiv%3A2602%2E03412%2E)
21. Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020a. [Mapping natural language instructions to mobile ui action sequences](https://md.research.elamur.ai/free/en/papers/li-2020-mapping-natural-language-instructions-mobile/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8198–8210.
22. Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020b. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprint arXiv:2010.04295. [find](https://scholar.google.com/scholar?q=Yang+Li%2C+Gang+Li%2C+Luheng+He%2C+Jingjie+Zheng%2C+Hong+Li%2C+and+Zhiwei+Guan%2E+2020b%2E+Widget+captioning%3A+Generating+natural+language+description+for+mobile+user+interface+elements%2E+arXiv+preprint+arXiv%3A2010%2E04)
23. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. Showui: One vision-language-action model for generalist gui agent. In NeurIPS 2024 Workshop on Open-World Agents, volume 1. [find](https://scholar.google.com/scholar?q=Kevin+Qinghong+Lin%2C+Linjie+Li%2C+Difei+Gao%2C+Zhengyuan+Yang%2C+Zechen+Bai%2C+Weixian+Lei%2C+Lijuan+Wang%2C+and+Mike+Zheng+Shou%2E+2024%2E+Showui%3A+One+vision%2Dlanguage%2Daction+model+for+generalist+gui+agent%2E+In+NeurIPS)
24. Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, and Yan Lu. 2025a. Ui-e2i-synth: Advancing gui grounding with large-scale instruction synthesis. arXiv preprint arXiv:2504.11257. [find](https://scholar.google.com/scholar?q=Xinyi+Liu%2C+Xiaoyi+Zhang%2C+Ziyun+Zhang%2C+and+Yan+Lu%2E+2025a%2E+Ui%2De2i%2Dsynth%3A+Advancing+gui+grounding+with+large%2Dscale+instruction+synthesis%2E+arXiv+preprint+arXiv%3A2504%2E11257%2E)
25. Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. 2025b. In-figuiagent: A multimodal generalist gui agent with native reasoning and reflection. arXiv preprint arXiv:2501.04575. — [arXiv:2501.04575](https://arxiv.org/abs/2501.04575) [find](https://scholar.google.com/scholar?q=Yuhang+Liu%2C+Pengxiang+Li%2C+Zishu+Wei%2C+Congkai+Xie%2C+Xueyu+Hu%2C+Xinchen+Xu%2C+Shengyu+Zhang%2C+Xiaotian+Han%2C+Hongxia+Yang%2C+and+Fei+Wu%2E+2025b%2E+In%2Dfiguiagent%3A+A+multimodal+generalist+gui+agent+with+native+reaso)
26. Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xavier Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, and 1 others. 2025c. Infigui-g1: Advancing gui grounding with adaptive exploration policy optimization. arXiv preprint arXiv:2508.05731. — [arXiv:2508.05731](https://arxiv.org/abs/2508.05731) [find](https://scholar.google.com/scholar?q=Yuhang+Liu%2C+Zeyu+Liu%2C+Shuanghe+Zhu%2C+Pengxiang+Li%2C+Congkai+Xie%2C+Jiasheng+Wang%2C+Xavier+Hu%2C+Xiaotian+Han%2C+Jianbo+Yuan%2C+Xinyao+Wang%2C+and+1+others%2E+2025c%2E+Infigui%2Dg1%3A+Advancing+gui+grounding+with+adaptive+)
27. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. 2025. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620. — [arXiv:2503.21620](https://arxiv.org/abs/2503.21620) [find](https://scholar.google.com/scholar?q=Zhengxi+Lu%2C+Yuxiang+Chai%2C+Yaxuan+Guo%2C+Xi+Yin%2C+Liang+Liu%2C+Hao+Wang%2C+Han+Xiao%2C+Shuai+Ren%2C+Guanjing+Xiong%2C+and+Hongsheng+Li%2E+2025%2E+Ui%2Dr1%3A+Enhancing+efficient+action+prediction+of+gui+agents+by+reinforcem)
28. Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. 2025. Visual test-time scaling for gui agent grounding. arXiv preprint arXiv:2505.00684. — [arXiv:2505.00684](https://arxiv.org/abs/2505.00684) [find](https://scholar.google.com/scholar?q=Tiange+Luo%2C+Lajanugen+Logeswaran%2C+Justin+Johnson%2C+and+Honglak+Lee%2E+2025%2E+Visual+test%2Dtime+scaling+for+gui+agent+grounding%2E+arXiv+preprint+arXiv%3A2505%2E00684%2E+%E2%80%94+%5BarXiv%3A2505%2E00684%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
29. Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, and 1 others. 2025. Ui-vision: A desktop-centric gui benchmark for visual perception and interaction. arXiv preprint arXiv:2503.15661. — [arXiv:2503.15661](https://arxiv.org/abs/2503.15661) [find](https://scholar.google.com/scholar?q=Shravan+Nayak%2C+Xiangru+Jian%2C+Kevin+Qinghong+Lin%2C+Juan+A+Rodriguez%2C+Montek+Kalsi%2C+Rabiul+Awal%2C+Nicolas+Chapados%2C+M+Tamer+%C3%96zsu%2C+Aishwarya+Agrawal%2C+David+Vazquez%2C+and+1+others%2E+2025%2E+Ui%2Dvision%3A+A+desktop)
30. Anthony Nguyen. 2024. Improved gui grounding via iterative narrowing. arXiv preprint arXiv:2411.13591. — [arXiv:2411.13591](https://arxiv.org/abs/2411.13591) [find](https://scholar.google.com/scholar?q=Anthony+Nguyen%2E+2024%2E+Improved+gui+grounding+via+iterative+narrowing%2E+arXiv+preprint+arXiv%3A2411%2E13591%2E+%E2%80%94+%5BarXiv%3A2411%2E13591%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2411%2E13591%29)
31. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. [Ui-tars: Pioneering automated gui interaction with native agents](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/). arXiv preprint arXiv:2501.12326.
32. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in neural information processing systems, 36:53728–53741.
33. Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, and 1 others. 2025. Gui-g 2 : Gaussian reward modeling for gui grounding. arXiv preprint arXiv:2507.15846. — [arXiv:2507.15846](https://arxiv.org/abs/2507.15846) [find](https://scholar.google.com/scholar?q=Fei+Tang%2C+Zhangxuan+Gu%2C+Zhengxi+Lu%2C+Xuyang+Liu%2C+Shuheng+Shen%2C+Changhua+Meng%2C+Wen+Wang%2C+Wenqi+Zhang%2C+Yongliang+Shen%2C+Weiming+Lu%2C+and+1+others%2E+2025%2E+Gui%2Dg+2+%3A+Gaussian+reward+modeling+for+gui+grounding)
34. Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, and 1 others. 2025. Mmbench-gui: Hierarchical multi-platform evaluation framework for gui agents. arXiv preprint arXiv:2507.19478. — [arXiv:2507.19478](https://arxiv.org/abs/2507.19478) [find](https://scholar.google.com/scholar?q=Xuehui+Wang%2C+Zhenyu+Wu%2C+JingJing+Xie%2C+Zichen+Ding%2C+Bowen+Yang%2C+Zehao+Li%2C+Zhaoyang+Liu%2C+Qingyun+Li%2C+Xuan+Dong%2C+Zhe+Chen%2C+and+1+others%2E+2025%2E+Mmbench%2Dgui%3A+Hierarchical+multi%2Dplatform+evaluation+framewor)
35. Yiqin Wang, Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual gui agent towards general computer control, 2024b. URL https://arxiv.org/abs/2412.01268. [↗](https://arxiv.org/abs/2412.01268)
36. Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang, and Yiwei Wang. 2025a. [Dimo-gui: Advancing test-time scaling in gui grounding via modality-aware visual reasoning](https://md.research.elamur.ai/free/en/papers/wu-2025-dimo-gui-advancing-test-time/). arXiv preprint arXiv:2507.00008.
37. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, and 1 others. 2025b. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143. — [arXiv:2506.03143](https://arxiv.org/abs/2506.03143) [find](https://scholar.google.com/scholar?q=Qianhui+Wu%2C+Kanzhi+Cheng%2C+Rui+Yang%2C+Chaoyun+Zhang%2C+Jianwei+Yang%2C+Huiqiang+Jiang%2C+Jian+Mu%2C+Baolin+Peng%2C+Bo+Qiao%2C+Reuben+Tan%2C+and+1+others%2E+2025b%2E+Gui%2Dactor%3A+Coordinate%2Dfree+visual+grounding+for+gui+age)
38. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and 1 others. 2024. [Os-atlas: A foundation action model for generalist gui agents](https://md.research.elamur.ai/free/en/papers/wu-2024-os-atlas-foundation-action-model/). arXiv preprint arXiv:2410.23218.
39. Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, and 1 others. 2025. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint arXiv:2505.13227. [find](https://scholar.google.com/scholar?q=Tianbao+Xie%2C+Jiaqi+Deng%2C+Xiaochuan+Li%2C+Junlin+Yang%2C+Haoyuan+Wu%2C+Jixuan+Chen%2C+Wenjing+Hu%2C+Xinyuan+Wang%2C+Yuhui+Xu%2C+Zekun+Wang%2C+and+1+others%2E+2025%2E+Scaling+computer%2Duse+grounding+via+user+interface+decom)
40. Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. 2025. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8851– 8859. [find](https://scholar.google.com/scholar?q=Hai%2DMing+Xu%2C+Qi+Chen%2C+Lei+Wang%2C+and+Lingqiao+Liu%2E+2025%2E+Attention%2Ddriven+gui+grounding%3A+Leveraging+pretrained+multimodal+large+language+models+without+fine%2Dtuning%2E+In+Proceedings+of+the+AAAI+Conferenc)
41. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. — [arXiv:2412.04454](https://arxiv.org/abs/2412.04454) [find](https://scholar.google.com/scholar?q=Yiheng+Xu%2C+Zekun+Wang%2C+Junli+Wang%2C+Dunjie+Lu%2C+Tianbao+Xie%2C+Amrita+Saha%2C+Doyen+Sahoo%2C+Tao+Yu%2C+and+Caiming+Xiong%2E+2024%2E+Aguvis%3A+Unified+pure+vision+agents+for+autonomous+gui+interaction%2E+arXiv+preprint+)
42. Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, and 1 others. 2025. Gta1: Gui test-time scaling agent. arXiv preprint arXiv:2507.05791. — [arXiv:2507.05791](https://arxiv.org/abs/2507.05791) [find](https://scholar.google.com/scholar?q=Yan+Yang%2C+Dongxu+Li%2C+Yutong+Dai%2C+Yuhao+Yang%2C+Ziyang+Luo%2C+Zirui+Zhao%2C+Zhiyuan+Hu%2C+Junzhe+Huang%2C+Amrita+Saha%2C+Zeyuan+Chen%2C+and+1+others%2E+2025%2E+Gta1%3A+Gui+test%2Dtime+scaling+agent%2E+arXiv+preprint+arXiv%3A250)
43. Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, and Dong Yu. 2025. Guided self-evolving llms with minimal human supervision. arXiv preprint arXiv:2512.02472. — [arXiv:2512.02472](https://arxiv.org/abs/2512.02472) [find](https://scholar.google.com/scholar?q=Wenhao+Yu%2C+Zhenwen+Liang%2C+Chengsong+Huang%2C+Kishan+Panaganti%2C+Tianqing+Fang%2C+Haitao+Mi%2C+and+Dong+Yu%2E+2025%2E+Guided+self%2Devolving+llms+with+minimal+human+supervision%2E+arXiv+preprint+arXiv%3A2512%2E02472%2E+%E2%80%94+%5B)
44. Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, and 1 others. Se-gui: Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Proc… — [arXiv:2411.18279](https://arxiv.org/abs/2411.18279) [find](https://scholar.google.com/scholar?q=Xinbin+Yuan%2C+Jian+Zhang%2C+Kaixin+Li%2C+Zhuoxuan+Cai%2C+Lujian+Yao%2C+Jie+Chen%2C+Enguang+Wang%2C+Qibin+Hou%2C+Jinwei+Chen%2C+Peng%2DTao+Jiang%2C+and+1+others%2E+Se%2Dgui%3A+Enhancing+visual+grounding+for+gui+agents+via+self%2De)
45. Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. 2025. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937. — [arXiv:2503.12937](https://arxiv.org/abs/2503.12937) [find](https://scholar.google.com/scholar?q=Jingyi+Zhang%2C+Jiaxing+Huang%2C+Huanjin+Yao%2C+Shunyu+Liu%2C+Xikun+Zhang%2C+Shijian+Lu%2C+and+Dacheng+Tao%2E+2025%2E+R1%2Dvl%3A+Learning+to+reason+with+multimodal+large+language+models+via+step%2Dwise+group+relative+polic)
46. Yu Zhao, Wei-Ning Chen, Huseyin Atahan Inan, Samuel Kessler, Lu Wang, Lukas Wutschitz, Fangkai Yang, Chaoyun Zhang, Pasquale Minervini, Saravan Rajmohan, and 1 others. 2025. Learning gui grounding with spatial reasoning from visual feedback. arXiv preprint arXiv:2509.21552. — [arXiv:2509.21552](https://arxiv.org/abs/2509.21552) [find](https://scholar.google.com/scholar?q=Yu+Zhao%2C+Wei%2DNing+Chen%2C+Huseyin+Atahan+Inan%2C+Samuel+Kessler%2C+Lu+Wang%2C+Lukas+Wutschitz%2C+Fangkai+Yang%2C+Chaoyun+Zhang%2C+Pasquale+Minervini%2C+Saravan+Rajmohan%2C+and+1+others%2E+2025%2E+Learning+gui+grounding+wit)

*Updated 1 November 2026*