---
title: "EVA — paper by Lu et al. (2026) — summary and ideas"
description: "Building on this, they propose EVA, a black-box evolutionary framework that mutates only the text of the payload. 54 references."
authors: ["Yijie Lu", "Manman Zhao", "Tianjie Ju", "Zihe Yan", "Xinbei Ma", "Yuan Guo", "Daizong Ding", "Gongshen Liu", "Zhuosheng Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1230/"
doi: "10.18653/v1/2026.findings-acl.1230"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2026-eva-evolving-semantic-adversaries-red/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2026-eva-evolving-semantic-adversaries-red/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks

# EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks

**Authors:** Yijie Lu, Manman Zhao, Tianjie Ju, Zihe Yan, [Xinbei Ma](https://md.research.elamur.ai/free/en/authors/xinbei-ma/), Yuan Guo, Daizong Ding, [Gongshen Liu](https://md.research.elamur.ai/free/en/authors/gongshen-liu/) and [Zhuosheng Zhang](https://md.research.elamur.ai/free/en/authors/zhuosheng-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1230/) · [PDF](https://aclanthology.org/2026.findings-acl.1230.pdf) · [DOI 10.18653/v1/2026.findings-acl.1230](https://doi.org/10.18653/v1/2026.findings-acl.1230) · [OpenAlex](https://openalex.org/W4415022989)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
GUI agents built on multimodal LLMs can be hijacked by Environmental Injection Attacks (EIAs), in which malicious instructions appear inside the interface, for example as fake pop-up notifications. Existing red-teaming approaches tend to be costly or inflexible. They also leave open whether attacks succeed because of how the injected element looks or because of what it says.

The authors ran a pilot study that changed pop-up position, size and colour while keeping the message fixed. Attack success barely moved, which points to semantic deception as the real bottleneck. Building on this, they propose EVA, a black-box evolutionary framework that mutates only the text of the payload. It works in two stages:
- **Offline discovery:** introspection-guided evolution mines linguistic weaknesses and condenses them into an interpretable rule library.
- **Online deployment:** the rules generate zero-shot attacks without any further search.

Across five victim agents, EVA reaches high attack success and converges in very few iterations. The authors argue that this exposes a dense semantic attack space and an "alignment paradox": obedience learned through alignment training makes agents trust content that sounds authoritative. They suggest defences based on checking semantic intent.

## Key points
- Average attack success rate ranges from 59% to 85% across five victim agents, covering both proprietary and open-source models. EVA beats the baselines.
- Benign seed messages become successful attacks after only 1.18–1.71 evolution iterations on average.
- The pilot study used 100 tasks sampled from the authors' manually collected EVA-GUI Benchmark. On Qwen2.5-VL-7B-Instruct and GPT-4V, ASR stayed within a narrow range regardless of visual variation.
- Embeddings of successful payloads cluster on a smooth, dense surface, suggesting the vulnerability is systematic rather than a set of isolated bugs. Pop-ups framed as mandatory blockers, such as security checks or required updates, are especially effective.
- Experiments ran on static local replicas of four web applications. A stated limitation is that the coarse feedback, which sorts failures into rejection or neglect, restricts how diverse the explored attacks can be.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 149 works · 16,161 citations
- [Wuhan University](https://md.research.elamur.ai/free/en/universities/wuhan-university/) — University, Wuhan, China · 35 works · 1,127 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works

🧭 Topics (1): [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Yijie Lu, Manman Zhao, Tianjie Ju, Zihe Yan, Xinbei Ma, Yuan Guo, Daizong Ding, Gongshen Liu and Zhuosheng Zhang. 2026. EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lu2026eva,
  title = {{EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks}},
  author = {Yijie Lu, Manman Zhao, Tianjie Ju, Zihe Yan, Xinbei Ma, Yuan Guo, Daizong Ding, Gongshen Liu and Zhuosheng Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1230},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1230}
}
```

- [← previous eSciBench: An Extensible Scientific PDF Extraction Benchmark](https://md.research.elamur.ai/free/en/papers/taillon-2026-escibench-extensible-scientific-pdf-extraction/)
- [next → Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring](https://md.research.elamur.ai/free/en/papers/kubesch-2026-evaluating-austrian-level-german-essays/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/)

## References

54 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv: 2303.08774.
2. Gabriel Alon and Michael Kamfonas. 2023. [Detecting language model attacks with perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/). Preprint, arXiv:2308.14132.
3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. [Qwen2.5-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/). Preprint, arXiv:2502.1…
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, and 32 others. 20… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
5. Chaoran Chen, Zhiping Zhang, Bingcan Guo, Shang Ma, Ibrahim Khalilov, Simret A Gebreegziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, and Toby Jia-Jun Li. 2025a. The obvious invisible threat: Llm-powered gui agents’ vulnerability to fine-print injections. Preprint, arXiv:2504.11281. — [arXiv:2504.11281](https://arxiv.org/abs/2504.11281) [find](https://scholar.google.com/scholar?q=Chaoran+Chen%2C+Zhiping+Zhang%2C+Bingcan+Guo%2C+Shang+Ma%2C+Ibrahim+Khalilov%2C+Simret+A+Gebreegziabher%2C+Yanfang+Ye%2C+Ziang+Xiao%2C+Yaxing+Yao%2C+Tianshi+Li%2C+and+Toby+Jia%2DJun+Li%2E+2025a%2E+The+obvious+invisible+threat%3A)
6. Yulin Chen, Haoran Li, Yuan Sui, Yufei He, Yue Liu, Yangqiu Song, and Bryan Hooi. 2025b. [Can indirect prompt injection attacks be detected and removed?](https://md.research.elamur.ai/free/en/papers/chen-2025-can-indirect-prompt-injection-attacks/) In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18189–18206, Vienna, Austr…
7. Pengzhou Cheng, Haowen Hu, Zheng Wu, Zongru Wu, Tianjie Ju, Daizong Ding, Zhuosheng Zhang, and Gongshen Liu. 2025. Hidden ghost hand: Unveiling backdoor vulnerabilities in MLLM-powered mobile GUI agents. In Findings of the Association 24605 9 for Computational Linguistics: EMNLP 2025, pages 7781–780… [find](https://scholar.google.com/scholar?q=Pengzhou+Cheng%2C+Haowen+Hu%2C+Zheng+Wu%2C+Zongru+Wu%2C+Tianjie+Ju%2C+Daizong+Ding%2C+Zhuosheng+Zhang%2C+and+Gongshen+Liu%2E+2025%2E+Hidden+ghost+hand%3A+Unveiling+backdoor+vulnerabilities+in+MLLM%2Dpowered+mobile+GUI+agen)
8. Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. 2025. Jail-breakRadar: Comprehensive assessment of jailbreak attacks against LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21538–21566,… [find](https://scholar.google.com/scholar?q=Junjie+Chu%2C+Yugeng+Liu%2C+Ziqing+Yang%2C+Xinyue+Shen%2C+Michael+Backes%2C+and+Yang+Zhang%2E+2025%2E+Jail%2DbreakRadar%3A+Comprehensive+assessment+of+jailbreak+attacks+against+LLMs%2E+In+Proceedings+of+the+63rd+Annual+M)
9. Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. 2024. On the robustness of large multimodal models against image adversarial attacks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24625–24634. [find](https://scholar.google.com/scholar?q=Xuanming+Cui%2C+Alejandro+Aparcedo%2C+Young+Kyun+Jang%2C+and+Ser%2DNam+Lim%2E+2024%2E+On+the+robustness+of+large+multimodal+models+against+image+adversarial+attacks%2E+In+2024+IEEE%2FCVF+Conference+on+Computer+Vision)
10. Xiaohu Du, Fan Mo, Ming Wen, Tu Gu, Huadi Zheng, Hai Jin, and Jie Shi. 2025. Multi-turn jailbreaking large language models via attention shifting. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Inte… [find](https://scholar.google.com/scholar?q=Xiaohu+Du%2C+Fan+Mo%2C+Ming+Wen%2C+Tu+Gu%2C+Huadi+Zheng%2C+Hai+Jin%2C+and+Jie+Shi%2E+2025%2E+Multi%2Dturn+jailbreaking+large+language+models+via+attention+shifting%2E+In+Proceedings+of+the+Thirty%2DNinth+AAAI+Conference+on)
11. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. [Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/). In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
12. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. [WebVoyager: Building an end-to-end web agent with large multimodal models](https://md.research.elamur.ai/free/en/papers/he-2024-webvoyager-building-end-end-web/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
13. Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, and 6 others. 2024. [Cogvlm2: Visual language models for image and video understa…](https://md.research.elamur.ai/free/en/papers/hong-2024-cogvlm2-visual-language-models/)
14. Feiran Jia, Tong Wu, Xin Qin, and Anna Squicciarini. 2025. [The task shield: Enforcing task alignment to defend against indirect prompt injection in LLM agents](https://md.research.elamur.ai/free/en/papers/jia-2025-task-shield-enforcing-task-alignment/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29680–29697, Vienn…
15. Yang JingYi, Shuai Shao, Dongrui Liu, and Jing Shao. 2025. RiOSWorld: Benchmarking the risk of multimodal computer-use agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yang+JingYi%2C+Shuai+Shao%2C+Dongrui+Liu%2C+and+Jing+Shao%2E+2025%2E+RiOSWorld%3A+Benchmarking+the+risk+of+multimodal+computer%2Duse+agents%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neural+Information+Processing+Sy)
16. Tianjie Ju, Yujia Chen, Hao Fei, Mong-Li Lee, Wynne Hsu, Pengzhou Cheng, Zongru Wu, Zhuosheng Zhang, and Gongshen Liu. 2025a. On the adaptive psychological persuasion of large language models. Preprint, arXiv:2506.06800. — [arXiv:2506.06800](https://arxiv.org/abs/2506.06800) [find](https://scholar.google.com/scholar?q=Tianjie+Ju%2C+Yujia+Chen%2C+Hao+Fei%2C+Mong%2DLi+Lee%2C+Wynne+Hsu%2C+Pengzhou+Cheng%2C+Zongru+Wu%2C+Zhuosheng+Zhang%2C+and+Gongshen+Liu%2E+2025a%2E+On+the+adaptive+psychological+persuasion+of+large+language+models%2E+Preprin)
17. Tianjie Ju, Yi Hua, Hao Fei, Zhenyu Shao, Yubin Zheng, Haodong Zhao, Mong-Li Lee, Wynne Hsu, Zhuosheng Zhang, and Gongshen Liu. 2025b. Watch out your album! on the inadvertent privacy memorization in multi-modal large language models. In Proceedings of the 42nd International Conference on Machine Le… [find](https://scholar.google.com/scholar?q=Tianjie+Ju%2C+Yi+Hua%2C+Hao+Fei%2C+Zhenyu+Shao%2C+Yubin+Zheng%2C+Haodong+Zhao%2C+Mong%2DLi+Lee%2C+Wynne+Hsu%2C+Zhuosheng+Zhang%2C+and+Gongshen+Liu%2E+2025b%2E+Watch+out+your+album%21+on+the+inadvertent+privacy+memorization+in+)
18. Tianjie Ju, Weiwei Sun, Wei Du, Xinwei Yuan, Zhaochun Ren, and Gongshen Liu. 2024. How large language models encode context knowledge? a layer-wise probing study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 20… [find](https://scholar.google.com/scholar?q=Tianjie+Ju%2C+Weiwei+Sun%2C+Wei+Du%2C+Xinwei+Yuan%2C+Zhaochun+Ren%2C+and+Gongshen+Liu%2E+2024%2E+How+large+language+models+encode+context+knowledge%3F+a+layer%2Dwise+probing+study%2E+In+Proceedings+of+the+2024+Joint+Inte)
19. Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Elaine T Chang, Vaughn Robinson, Shuyan Zhou, Matt Fredrikson, Sean M. Hendryx, Summer Yue, and Zifan Wang. 2025. Aligned LLMs are not aligned browser agents. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Priyanshu+Kumar%2C+Elaine+Lau%2C+Saranya+Vijayakumar%2C+Tu+Trinh%2C+Elaine+T+Chang%2C+Vaughn+Robinson%2C+Shuyan+Zhou%2C+Matt+Fredrikson%2C+Sean+M%2E+Hendryx%2C+Summer+Yue%2C+and+Zifan+Wang%2E+2025%2E+Aligned+LLMs+are+not+align)
20. Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, J Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. 2025. OS-harm: A benchmark for measuring safety of computer use agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Thomas+Kuntz%2C+Agatha+Duzan%2C+Hao+Zhao%2C+Francesco+Croce%2C+J+Zico+Kolter%2C+Nicolas+Flammarion%2C+and+Maksym+Andriushchenko%2E+2025%2E+OS%2Dharm%3A+A+benchmark+for+measuring+safety+of+computer+use+agents%2E+In+The+Thir)
21. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In Proceedings of the 41st Internatio… [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Ren+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+2024%2E+RLAIF+vs%2E+RLHF%3A+Scalin)
22. Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: screenshot parsing as pretraining for visual language understanding. In Proceedings of the 40th International Conference on Mach… [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Mandar+Joshi%2C+Iulia+Turc%2C+Hexiang+Hu%2C+Fangyu+Liu%2C+Julian+Eisenschlos%2C+Urvashi+Khandelwal%2C+Peter+Shaw%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2023%2E+Pix2struct%3A+screenshot+parsing+as+pretra)
23. Ning Li, Qiqiang Lin, Zheng Wu, Xiaoyun Mo, Weiming Zhang, Yin Zhao, Xiangmou Qu, Jiamu Zhou, Jun Wang, Congmin Zheng, Yuanyi Song, Hongjiang Chen, Heyuan Huang, Jihong Wang, Jiaxin Yin, Jingwei Yu, Junwei Liao, Qiuying Peng, Xingyu Lou, and 4 others. 2025. Coloragent: Building a robust, 24606 10 pe… — [arXiv:2510.19386](https://arxiv.org/abs/2510.19386) [find](https://scholar.google.com/scholar?q=Ning+Li%2C+Qiqiang+Lin%2C+Zheng+Wu%2C+Xiaoyun+Mo%2C+Weiming+Zhang%2C+Yin+Zhao%2C+Xiangmou+Qu%2C+Jiamu+Zhou%2C+Jun+Wang%2C+Congmin+Zheng%2C+Yuanyi+Song%2C+Hongjiang+Chen%2C+Heyuan+Huang%2C+Jihong+Wang%2C+Jiaxin+Yin%2C+Jingwei+Yu%2C+J)
24. Preprint, Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2025. EIA: ENVIRONMENTAL INJEC-TION ATTACK ON GENERALIST WEB AGENTS FOR PRIVACY LEAKAGE. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Preprint%2C+Zeyi+Liao%2C+Lingbo+Mo%2C+Chejian+Xu%2C+Mintong+Kang%2C+Jiawei+Zhang%2C+Chaowei+Xiao%2C+Yuan+Tian%2C+Bo+Li%2C+and+Huan+Sun%2E+2025%2E+EIA%3A+ENVIRONMENTAL+INJEC%2DTION+ATTACK+ON+GENERALIST+WEB+AGENTS+FOR+PRIVACY+LE)
25. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. Preprint, arXiv:2408.00203. — [arXiv:2408.00203](https://arxiv.org/abs/2408.00203) [find](https://scholar.google.com/scholar?q=Yadong+Lu%2C+Jianwei+Yang%2C+Yelong+Shen%2C+and+Ahmed+Awadallah%2E+2024%2E+Omniparser+for+pure+vision+based+gui+agent%2E+Preprint%2C+arXiv%3A2408%2E00203%2E+%E2%80%94+%5BarXiv%3A2408%2E00203%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E00203%29)
26. Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. 2025. Caution for the environment: Multimodal LLM agents are susceptible to environmental distractions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Xinbei+Ma%2C+Yiting+Wang%2C+Yao+Yao%2C+Tongxin+Yuan%2C+Aston+Zhang%2C+Zhuosheng+Zhang%2C+and+Hai+Zhao%2E+2025%2E+Caution+for+the+environment%3A+Multimodal+LLM+agents+are+susceptible+to+environmental+distractions%2E+In+Pr)
27. Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Ariu. 2024. Filtered direct preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22729–22770, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tetsuro+Morimura%2C+Mitsuki+Sakamoto%2C+Yuu+Jinnai%2C+Kenshi+Abe%2C+and+Kaito+Ariu%2E+2024%2E+Filtered+direct+preference+optimization%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Languag)
28. Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, and 11 others. 2025. GUI agents: A survey. In Findings of the Associatio… [find](https://scholar.google.com/scholar?q=Dang+Nguyen%2C+Jian+Chen%2C+Yu+Wang%2C+Gang+Wu%2C+Namyong+Park%2C+Zhengmian+Hu%2C+Hanjia+Lyu%2C+Junda+Wu%2C+Ryan+Aponte%2C+Yu+Xia%2C+Xintong+Li%2C+Jing+Shi%2C+Hongjie+Chen%2C+Viet+Dac+Lai%2C+Zhouhang+Xie%2C+Sungchul+Kim%2C+Ruiyi+Zha)
29. OpenAI. 2025. Gpt-5 system card. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Gpt%2D5+system+card%2E)
30. OpenAI. 2025. Introducing Operator. [https://openai.com/index/introducing-operator/](https://openai.com/index/introducing-operator/). [↗](https://openai.com/index/introducing-operator/)
31. Lihang Pan, Chun Yu, Zhe He, and Yuanchun Shi. 2023. A human-computer collaborative editing tool for conceptual diagrams. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Lihang+Pan%2C+Chun+Yu%2C+Zhe+He%2C+and+Yuanchun+Shi%2E+2023%2E+A+human%2Dcomputer+collaborative+editing+tool+for+conceptual+diagrams%2E+In+Proceedings+of+the+2023+CHI+Conference+on+Human+Factors+in+Computing+System)
32. Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. 2025. Advprompter: Fast adaptive adversarial prompting for LLMs. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Anselm+Paulus%2C+Arman+Zharmagambetov%2C+Chuan+Guo%2C+Brandon+Amos%2C+and+Yuandong+Tian%2E+2025%2E+Advprompter%3A+Fast+adaptive+adversarial+prompting+for+LLMs%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Le)
33. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, and 16 others. 2025. [Ui-tars: Pioneering automated gui interaction with…](https://md.research.elamur.ai/free/en/papers/qin-2025-ui-tars-pioneering-automated/)
34. Aref Miri Rekavandi, Farhad Farokhi, Olga Ohrimenko, and Benjamin I. P. Rubinstein. 2024. Certified adversarial robustness via randomized $\\alpha$-smoothing for regression models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Aref+Miri+Rekavandi%2C+Farhad+Farokhi%2C+Olga+Ohrimenko%2C+and+Benjamin+I%2E+P%2E+Rubinstein%2E+2024%2E+Certified+adversarial+robustness+via+randomized+%24%5C%5Calpha%24%2Dsmoothing+for+regression+models%2E+In+The+Thirty%2Deight)
35. Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2025. Great, now write an article about that: the crescendo multi-turn llm jailbreak attack. In Proceedings of the 34th USENIX Conference on Security Symposium, SEC ’25, USA. USENIX Association. [find](https://scholar.google.com/scholar?q=Mark+Russinovich%2C+Ahmed+Salem%2C+and+Ronen+Eldan%2E+2025%2E+Great%2C+now+write+an+article+about+that%3A+the+crescendo+multi%2Dturn+llm+jailbreak+attack%2E+In+Proceedings+of+the+34th+USENIX+Conference+on+Security+Sy)
36. Kunal Singh, Shreyas Singh, and Mukund Khanna. 2025. Trishul: Towards Region Identification and Screen Hierarchy Understanding for Large VLM Based GUI Agents . In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 170–179, Los Alamitos, CA, USA. IEEE Compute… [find](https://scholar.google.com/scholar?q=Kunal+Singh%2C+Shreyas+Singh%2C+and+Mukund+Khanna%2E+2025%2E+Trishul%3A+Towards+Region+Identification+and+Screen+Hierarchy+Understanding+for+Large+VLM+Based+GUI+Agents+%2E+In+2025+IEEE%2FCVF+Conference+on+Computer+)
37. Yu Wang, Xiaofei Zhou, Yichen Wang, Geyuan Zhang, and Tianxing He. 2025. [Jailbreak large vision-language models through multi-modal linkage](https://md.research.elamur.ai/free/en/papers/wang-2025-jailbreak-large-vision-language-models/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1466–1494, Vienna, Austria. Associati…
38. Zixuan Weng, Xiaolong Jin, Jinyuan Jia, and Xiangyu Zhang. 2025. Foot-in-the-door: A multi-turn jailbreak for LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1939–1950, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zixuan+Weng%2C+Xiaolong+Jin%2C+Jinyuan+Jia%2C+and+Xiangyu+Zhang%2E+2025%2E+Foot%2Din%2Dthe%2Ddoor%3A+A+multi%2Dturn+jailbreak+for+LLMs%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
39. Chen Henry Wu, Rishi Rajesh Shah, Jing Yu Koh, Russ Salakhutdinov, Daniel Fried, and Aditi Raghunathan. 2025a. Dissecting adversarial robustness of multimodal LM agents. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chen+Henry+Wu%2C+Rishi+Rajesh+Shah%2C+Jing+Yu+Koh%2C+Russ+Salakhutdinov%2C+Daniel+Fried%2C+and+Aditi+Raghunathan%2E+2025a%2E+Dissecting+adversarial+robustness+of+multimodal+LM+agents%2E+In+The+Thirteenth+Internationa)
40. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, and Jianfeng Gao. 2025b. GUI-actor: Coordinate-free visual grounding for GUI agents. In… [find](https://scholar.google.com/scholar?q=Qianhui+Wu%2C+Kanzhi+Cheng%2C+Rui+Yang%2C+Chaoyun+Zhang%2C+Jianwei+Yang%2C+Huiqiang+Jiang%2C+Jian+Mu%2C+Baolin+Peng%2C+Bo+Qiao%2C+Reuben+Tan%2C+Si+Qin%2C+Lars+Liden%2C+Qingwei+Lin%2C+Huan+Zhang%2C+Tong+Zhang%2C+Jianbing+Zhang%2C+Don)
41. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. 2025. Advagent: Controllable blackbox red-teaming on web agents. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Chejian+Xu%2C+Mintong+Kang%2C+Jiawei+Zhang%2C+Zeyi+Liao%2C+Lingbo+Mo%2C+Mengqi+Yuan%2C+Huan+Sun%2C+and+Bo+Li%2E+2025%2E+Advagent%3A+Controllable+blackbox+red%2Dteaming+on+web+agents%2E+In+Forty%2Dsecond+International+Conferenc)
42. Zihe Yan, Kai Luo, Haoyu Yang, Yang Yu, Zhuosheng Zhang, and Guancheng Li. 2026. An llm-based quantitative framework for evaluating high-stealthy backdoor risks in oss supply chains. In Proceedings of the 40th AAAI Conference on Artificial Intelligence, volume 40, pages 1418–1425. AAAI Press. [find](https://scholar.google.com/scholar?q=Zihe+Yan%2C+Kai+Luo%2C+Haoyu+Yang%2C+Yang+Yu%2C+Zhuosheng+Zhang%2C+and+Guancheng+Li%2E+2026%2E+An+llm%2Dbased+quantitative+framework+for+evaluating+high%2Dstealthy+backdoor+risks+in+oss+supply+chains%2E+In+Proceedings+of)
43. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark 24607 11 prompting unleashes extraordinary visual grounding in gpt-4v. Preprint, arXiv:2310.11441. [find](https://scholar.google.com/scholar?q=Jianwei+Yang%2C+Hao+Zhang%2C+Feng+Li%2C+Xueyan+Zou%2C+Chunyuan+Li%2C+and+Jianfeng+Gao%2E+2023%2E+Set%2Dof%2Dmark+24607+11+prompting+unleashes+extraordinary+visual+grounding+in+gpt%2D4v%2E+Preprint%2C+arXiv%3A2310%2E11441%2E)
44. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. 2025. Mobile-agent-v3: Fundamental agents for gui automation. Preprint, arXiv:2508.15144. — [arXiv:2508.15144](https://arxiv.org/abs/2508.15144) [find](https://scholar.google.com/scholar?q=Jiabo+Ye%2C+Xi+Zhang%2C+Haiyang+Xu%2C+Haowei+Liu%2C+Junyang+Wang%2C+Zhaoqing+Zhu%2C+Ziwei+Zheng%2C+Feiyu+Gao%2C+Junjie+Cao%2C+Zhengxi+Lu%2C+Jitong+Liao%2C+Qi+Zheng%2C+Fei+Huang%2C+Jingren+Zhou%2C+and+Ming+Yan%2E+2025%2E+Mobile%2Dagent)
45. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024. [R-judge: Benchmarking safety risk awareness for LLM agents](https://md.research.elamur.ai/free/en/papers/yuan-2024-r-judge-benchmarking-safety-risk/). In Findings of the Association for Computational Linguistics: EMNLP 2024, pa…
46. Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. [find](https://scholar.google.com/scholar?q=Yongcheng+Zeng%2C+Guoqing+Liu%2C+Weiyu+Ma%2C+Ning+Yang%2C+Haifeng+Zhang%2C+and+Jun+Wang%2E+2024%2E+Token%2Dlevel+direct+preference+optimization%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning)
47. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. [InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents](https://md.research.elamur.ai/free/en/papers/zhan-2024-injecagent-benchmarking-indirect-prompt-injections/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 10471–10506, Bangkok, Thailand. Association for Computation…
48. Peng-Fei Zhang, Zi Huang, and Guangdong Bai. 2024. Universal adversarial perturbations for vision-language pre-trained models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 862–871, New York, NY, USA. Association f… [find](https://scholar.google.com/scholar?q=Peng%2DFei+Zhang%2C+Zi+Huang%2C+and+Guangdong+Bai%2E+2024%2E+Universal+adversarial+perturbations+for+vision%2Dlanguage+pre%2Dtrained+models%2E+In+Proceedings+of+the+47th+International+ACM+SIGIR+Conference+on+Research)
49. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2025. [Attacking vision-language computer agents via pop-ups](https://md.research.elamur.ai/free/en/papers/zhang-2025-attacking-vision-language-computer-agents/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8387–8401, Vienna, Austria. Association for Computational Linguistics.
50. Yihao Zhang and Zeming Wei. 2025. Boosting jailbreak attack with momentum. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. [find](https://scholar.google.com/scholar?q=Yihao+Zhang+and+Zeming+Wei%2E+2025%2E+Boosting+jailbreak+attack+with+momentum%2E+In+ICASSP+2025+%2D+2025+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+1%E2%80%935%2E)
51. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. [Gpt-4v(ision) is a generalist web agent, if grounded](https://md.research.elamur.ai/free/en/papers/zheng-2024-gpt-4v-ision-generalist-web/). In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
52. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2023. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In Second Agent Learning in Open-Endedness Workshop.
53. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. AutoDAN: Automatic and interpretable adversarial attacks on large language models. In Socially Responsible Language Modelling Research. [find](https://scholar.google.com/scholar?q=Sicheng+Zhu%2C+Ruiyi+Zhang%2C+Bang+An%2C+Gang+Wu%2C+Joe+Barrow%2C+Zichao+Wang%2C+Furong+Huang%2C+Ani+Nenkova%2C+and+Tong+Sun%2E+2023%2E+AutoDAN%3A+Automatic+and+interpretable+adversarial+attacks+on+large+language+models%2E+I)
54. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). Preprint, arXiv:2307.15043.

*Updated 1 November 2026*