---
title: "InjecAgent — paper by Zhan et al. (2024) — summary and ideas"
description: "LLM agents that call tools often read outside content such as emails, reviews, notes or web pages. 58 references, cited by 63 papers here."
authors: ["Qiusi Zhan", "Zhixiang Liang", "Zifan Ying", "Daniel Kang"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.624/"
doi: "10.18653/v1/2024.findings-acl.624"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhan-2024-injecagent-benchmarking-indirect-prompt-injections/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhan-2024-injecagent-benchmarking-indirect-prompt-injections/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

# InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

**Authors:** Qiusi Zhan, Zhixiang Liang, Zifan Ying and [Daniel Kang](https://md.research.elamur.ai/free/en/authors/daniel-kang/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.624/) · [PDF](https://aclanthology.org/2024.findings-acl.624.pdf) · [DOI 10.18653/v1/2024.findings-acl.624](https://doi.org/10.18653/v1/2024.findings-acl.624) · [OpenAlex](https://openalex.org/W4402670146)

**Citations:** 124 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/zhan-2024-injecagent-benchmarking-indirect-prompt-injections/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LLM agents that call tools often read outside content such as emails, reviews, notes or web pages. An attacker can plant instructions in that content to steer the agent against its own user. This is called indirect prompt injection (IPI). InjecAgent is presented as the first benchmark that measures how vulnerable tool-integrated agents are to IPI.

The authors started from the tool definitions of Ruan et al. (2023). They used GPT-4, followed by manual correction, to build 1,054 test cases. Each case pairs a user request with an attacker instruction hidden inside a tool response. Attacks either cause direct harm (for example, moving money or unlocking doors) or steal private data and email it to the attacker.

Across 30 agents, ReAct-prompted models proved easy to manipulate. Adding a fixed hacking prompt to the injected text pushed success rates higher. Agents fine-tuned for tool calling resisted noticeably better. The results suggest that agents with access to personal data and real-world actions need dedicated defenses before wide deployment.

## Key points
- The benchmark covers 1,054 cases, built from 17 user tools and 62 attacker tools. These were drawn from 330 tools across 36 toolkits (Ruan et al., 2023), spanning domains such as finance, smart home, email and health.
- GPT-4 generated the test cases in stages: user case templates with an attacker-instruction placeholder, then attacker cases. About 30% of the generated attacker instructions lacked required parameters and were manually fixed.
- ReAct-prompted GPT-4 followed injected instructions in 24% of cases. With the hacking prompt, this rose to 47%.
- Fine-tuned GPT-4 had an attack success rate of only 7.1%. Fine-tuned agents also produced valid outputs more often.
- Stated limitations include:
  - only one fixed hacking prompt was tested;
  - only single-turn scenarios with attacks of at most two steps;
  - injected text was never mixed with benign content;
  - only two fine-tuned agents were studied.
- The findings were disclosed to OpenAI and Anthropic.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 108 works · 67,432 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Qiusi Zhan, Zhixiang Liang, Zifan Ying and Daniel Kang. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhan2024injecagent,
  title = {{InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents}},
  author = {Qiusi Zhan, Zhixiang Liang, Zifan Ying and Daniel Kang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.624},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.624}
}
```

- [← previous Refusal in Language Models Is Mediated by a Single Direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/)
- [next → The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

58 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. [Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/). In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
3. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. [Do as i can, not as i say: Grounding language in robotic affordances](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/). arXiv preprint arXiv:2204.01691. [https://www.anthropic.com/](https://www.anthropic.com/) [↗](https://www.anthropic.com/)
4. Anthropic. 2023. news/claude-2. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+news%2Fclaude%2D2%2E)
5. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
7. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363. — [arXiv:2402.06363](https://arxiv.org/abs/2402.06363) [find](https://scholar.google.com/scholar?q=Sizhe+Chen%2C+Julien+Piet%2C+Chawin+Sitawarin%2C+and+David+Wagner%2E+2024%2E+Struq%3A+Defending+against+prompt+injection+with+structured+queries%2E+arXiv+preprint+arXiv%3A2402%2E06363%2E+%E2%80%94+%5BarXiv%3A2402%2E06363%5D%28https%3A%2F%2Farxi)
8. Harald Cramér. 1999. Mathematical methods of statistics, volume 26. Princeton university press. [find](https://scholar.google.com/scholar?q=Harald+Cram%C3%A9r%2E+1999%2E+Mathematical+methods+of+statistics%2C+volume+26%2E+Princeton+university+press%2E)
9. Luigi Daniele and Suphavadeeprasit. 2023. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(comming soon). [find](https://scholar.google.com/scholar?q=Luigi+Daniele+and+Suphavadeeprasit%2E+2023%2E+Amplify%2Dinstruct%3A+Synthetically+generated+diverse+multi%2Dturn+conversations+for+effecient+llm+training%2E+arXiv+preprint+arXiv%3A%28comming+soon%29%2E)
10. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. [Mind2web: Towards a generalist agent for the web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/). arXiv preprint arXiv:2306.06070.
11. Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American statistical association, 56(293):52–64. [find](https://scholar.google.com/scholar?q=Olive+Jean+Dunn%2E+1961%2E+Multiple+comparisons+among+means%2E+Journal+of+the+American+statistical+association%2C+56%28293%29%3A52%E2%80%9364%2E)
12. Aysan Esmradi, Daniel Wankit Yip, and Chun Fai Chan. 2023. A comprehensive survey of attack techniques, implementation, and mitigation strategies in large language models. arXiv preprint arXiv:2312.10982. — [arXiv:2312.10982](https://arxiv.org/abs/2312.10982) [find](https://scholar.google.com/scholar?q=Aysan+Esmradi%2C+Daniel+Wankit+Yip%2C+and+Chun+Fai+Chan%2E+2023%2E+A+comprehensive+survey+of+attack+techniques%2C+implementation%2C+and+mitigation+strategies+in+large+language+models%2E+arXiv+preprint+arXiv%3A2312%2E10)
13. Kai Greshake. 2023. Inject my pdf: Prompt injection for your resume. [find](https://scholar.google.com/scholar?q=Kai+Greshake%2E+2023%2E+Inject+my+pdf%3A+Prompt+injection+for+your+resume%2E)
14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023a%2E+Mis)
15. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023b. [Llm-blender: Ensembling large language models with pairwise ranking and generative fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/).
16. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733. — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) [find](https://scholar.google.com/scholar?q=Daniel+Kang%2C+Xuechen+Li%2C+Ion+Stoica%2C+Carlos+Guestrin%2C+Matei+Zaharia%2C+and+Tatsunori+Hashimoto%2E+2023%2E+Exploiting+programmatic+behavior+of+llms%3A+Dual%2Duse+through+standard+security+attacks%2E+arXiv+preprint)
17. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023. Platypus: Quick, cheap, and powerful refinement of llms. [find](https://scholar.google.com/scholar?q=Ariel+N%2E+Lee%2C+Cole+J%2E+Hunter%2C+and+Nataniel+Ruiz%2E+2023%2E+Platypus%3A+Quick%2C+cheap%2C+and+powerful+refinement+of+llms%2E)
18. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023. [Prompt injection attack against llm-integrated applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/). arXiv preprint arXiv:2306.05499.
19. Pattie Maes. 1995. Agents that reduce work and information overload. In Readings in human–computer interaction, pages 811–821. Elsevier. [find](https://scholar.google.com/scholar?q=Pattie+Maes%2E+1995%2E+Agents+that+reduce+work+and+information+overload%2E+In+Readings+in+human%E2%80%93computer+interaction%2C+pages+811%E2%80%93821%2E+Elsevier%2E)
20. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/).
21. NVIDIA. 2023. Securing LLM Systems Against Prompt Injection \| NVIDIA Technical Blog — developer.nvidia.com. [https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection/](https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection/). [↗](https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection/)
22. NVIDIA. 2024. NVIDIA Chat With RTX — nvidia.com. [https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/](https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/). https://openai.com/blog/ [↗](https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/)
23. OpenAI. 2023a. chatgpt-plugins. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+chatgpt%2Dplugins%2E)
24. OpenAI. 2023b. [https://platform.openai.com/docs/guides/function-calling/supported-models](https://platform.openai.com/docs/guides/function-calling/supported-models). [↗](https://platform.openai.com/docs/guides/function-calling/supported-models)
25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155.
26. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). arXiv preprint arXiv:2305.15334.
27. Fábio Perez and Ian Ribeiro. 2022. [Ignore previous prompt: Attack techniques for language models](https://md.research.elamur.ai/free/en/papers/perez-2022-ignore-previous-prompt-attack-techniques/). arXiv preprint arXiv:2211.09527.
28. Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2023. Jatmo: Prompt injection defense by task-specific finetuning. arXiv preprint arXiv:2312.17673. — [arXiv:2312.17673](https://arxiv.org/abs/2312.17673) [find](https://scholar.google.com/scholar?q=Julien+Piet%2C+Maha+Alrashed%2C+Chawin+Sitawarin%2C+Sizhe+Chen%2C+Zeming+Wei%2C+Elizabeth+Sun%2C+Basel+Alomair%2C+and+David+Wagner%2E+2023%2E+Jatmo%3A+Prompt+injection+defense+by+task%2Dspecific+finetuning%2E+arXiv+preprint+)
29. Avram Piltch. 2023. Chatgpt plugins open security holes from pdfs, websites and more. [find](https://scholar.google.com/scholar?q=Avram+Piltch%2E+2023%2E+Chatgpt+plugins+open+security+holes+from+pdfs%2C+websites+and+more%2E)
30. Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. 2023. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models. [find](https://scholar.google.com/scholar?q=Michael+Poli%2C+Jue+Wang%2C+Stefano+Massaroli%2C+Jeffrey+Quesnelle%2C+Ryan+Carlow%2C+Eric+Nguyen%2C+and+Armin+Thomas%2E+2023%2E+StripedHyena%3A+Moving+Beyond+Transformers+with+Hybrid+Signal+Processing+Models%2E)
31. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. [Toolllm: Facilitating large language models to master 16000+ real-world apis](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/). arXiv preprint arXiv:2307.16789.
32. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/).
33. Johann Rehberger. 2023a. Indirect prompt injection via youtube transcripts. [find](https://scholar.google.com/scholar?q=Johann+Rehberger%2E+2023a%2E+Indirect+prompt+injection+via+youtube+transcripts%2E)
34. Johann Rehberger. 2023b. Openai removes the "chat with code" plugin from store. [find](https://scholar.google.com/scholar?q=Johann+Rehberger%2E+2023b%2E+Openai+removes+the+%22chat+with+code%22+plugin+from+store%2E)
35. Johann Rehberger. 2023c. Plugin vulnerabilities: Visit a website and have your source code stolen. [find](https://scholar.google.com/scholar?q=Johann+Rehberger%2E+2023c%2E+Plugin+vulnerabilities%3A+Visit+a+website+and+have+your+source+code+stolen%2E)
36. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. [Smoothllm: Defending large language models against jailbreaking attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/). arXiv preprint arXiv:2310.03684.
37. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817. [find](https://scholar.google.com/scholar?q=Yangjun+Ruan%2C+Honghua+Dong%2C+Andrew+Wang%2C+Silviu+Pitis%2C+Yongchao+Zhou%2C+Jimmy+Ba%2C+Yann+Dubois%2C+Chris+J+Maddison%2C+and+Tatsunori+Hashimoto%2E+2023%2E+Identifying+the+risks+of+lm+agents+with+an+lm%2Demulated+san)
38. Stuart J Russell and Peter Norvig. 2010. Artificial intelligence a modern approach. London. [find](https://scholar.google.com/scholar?q=Stuart+J+Russell+and+Peter+Norvig%2E+2010%2E+Artificial+intelligence+a+modern+approach%2E+London%2E)
39. Roman Samoilenko. 2023. New prompt injection attack on chatgpt web version. markdown images can steal your chat data. [find](https://scholar.google.com/scholar?q=Roman+Samoilenko%2E+2023%2E+New+prompt+injection+attack+on+chatgpt+web+version%2E+markdown+images+can+steal+your+chat+data%2E)
40. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). arXiv preprint arXiv:2302.04761.
41. Jose Selvi. 2022. Exploring prompt injection attacks. [https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/](https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/). [↗](https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/)
42. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. [Llm-planner: Few-shot grounded planning for embodied agents with large language models](https://md.research.elamur.ai/free/en/papers/song-2022-llm-planner-few-shot-grounded/). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3009.
43. Austin Stubbs. 2023. LLM Hacking: Prompt Injection Techniques — austin-stubbs. [https://medium.com/@austin-stubbs/llm-security-types-of-prompt-injection-d7ad8d7d75a3](https://medium.com/@austin-stubbs/llm-security-types-of-prompt-injection-d7ad8d7d75a3). [↗](https://medium.com/@austin-stubbs/llm-security-types-of-prompt-injection-d7ad8d7d75a3)
44. Xuchen Suo. 2024. Signed-prompt: A new approach to prevent prompt injection attacks against llm-integrated applications. arXiv preprint arXiv:2401.07612. — [arXiv:2401.07612](https://arxiv.org/abs/2401.07612) [find](https://scholar.google.com/scholar?q=Xuchen+Suo%2E+2024%2E+Signed%2Dprompt%3A+A+new+approach+to+prevent+prompt+injection+attacks+against+llm%2Dintegrated+applications%2E+arXiv+preprint+arXiv%3A2401%2E07612%2E+%E2%80%94+%5BarXiv%3A2401%2E07612%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F240)
45. Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855. — [arXiv:2311.11855](https://arxiv.org/abs/2311.11855) [find](https://scholar.google.com/scholar?q=Yu+Tian%2C+Xiao+Yang%2C+Jingyuan+Zhang%2C+Yinpeng+Dong%2C+and+Hang+Su%2E+2023%2E+Evil+geniuses%3A+Delving+into+the+safety+of+llm%2Dbased+agents%2E+arXiv+preprint+arXiv%3A2311%2E11855%2E+%E2%80%94+%5BarXiv%3A2311%2E11855%5D%28https%3A%2F%2Farxiv%2Eorg)
46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
47. Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. 2023. Tensor trust: Interpretable prompt injection attacks from an online game. arXiv preprint arXiv:2311.01011. — [arXiv:2311.01011](https://arxiv.org/abs/2311.01011) [find](https://scholar.google.com/scholar?q=Sam+Toyer%2C+Olivia+Watkins%2C+Ethan+Adrian+Mendes%2C+Justin+Svegliato%2C+Luke+Bailey%2C+Tiffany+Wang%2C+Isaac+Ong%2C+Karim+Elmaaroufi%2C+Pieter+Abbeel%2C+Trevor+Darrell%2C+et+al%2E+2023%2E+Tensor+trust%3A+Interpretable+prompt)
48. Lilian Weng. 2023. Llm powered autonomous agents. [find](https://scholar.google.com/scholar?q=Lilian+Weng%2E+2023%2E+Llm+powered+autonomous+agents%2E)
49. Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics: Methodology and Distribution, pages 196–202. Springer. [find](https://scholar.google.com/scholar?q=Frank+Wilcoxon%2E+1992%2E+Individual+comparisons+by+ranking+methods%2E+In+Breakthroughs+in+Statistics%3A+Methodology+and+Distribution%2C+pages+196%E2%80%93202%2E+Springer%2E)
50. Michael Wooldridge and Nicholas R Jennings. 1995. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115–152. [find](https://scholar.google.com/scholar?q=Michael+Wooldridge+and+Nicholas+R+Jennings%2E+1995%2E+Intelligent+agents%3A+Theory+and+practice%2E+The+knowledge+engineering+review%2C+10%282%29%3A115%E2%80%93152%2E)
51. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. [Wizardlm: Empowering large language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). arXiv preprint arXiv:2304.12244.
52. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629.
53. Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) [find](https://scholar.google.com/scholar?q=Jingwei+Yi%2C+Yueqi+Xie%2C+Bin+Zhu%2C+Keegan+Hines%2C+Emre+Kiciman%2C+Guangzhong+Sun%2C+Xing+Xie%2C+and+Fangzhao+Wu%2E+2023%2E+Benchmarking+and+defending+against+indirect+prompt+injection+attacks+on+large+language+mode)
54. Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, and Xinyu Xing. 2023. Assessing prompt injection risks in 200+ custom gpts. arXiv preprint arXiv:2311.11538. — [arXiv:2311.11538](https://arxiv.org/abs/2311.11538) [find](https://scholar.google.com/scholar?q=Jiahao+Yu%2C+Yuhang+Wu%2C+Dong+Shu%2C+Mingyu+Jin%2C+and+Xinyu+Xing%2E+2023%2E+Assessing+prompt+injection+risks+in+200%2B+custom+gpts%2E+arXiv+preprint+arXiv%3A2311%2E11538%2E+%E2%80%94+%5BarXiv%3A2311%2E11538%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311)
55. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. [Mmmu: A massive multi-disc…](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
56. Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Wenhu Chen, and Jie Fu. 2024. Cmmmu: A… [find](https://scholar.google.com/scholar?q=Ge+Zhang%2C+Xinrun+Du%2C+Bei+Chen%2C+Yiming+Liang%2C+Tongxu+Luo%2C+Tianyu+Zheng%2C+Kang+Zhu%2C+Yuyang+Cheng%2C+Chunpu+Xu%2C+Shuyue+Guo%2C+Haoran+Zhang%2C+Xingwei+Qu%2C+Junjie+Wang%2C+Ruibin+Yuan%2C+Yizhi+Li%2C+Zekun+Wang%2C+Yudong+L)
57. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)
58. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043.

*Updated 24 October 2026*