---
title: "Formalizing and Benchmarking Prompt Injection Attacks and"
description: "LLM-Integrated Applications build their prompts from an instruction and from data taken from outside… 42 references, cited by 71 papers here."
authors: ["Yupei Liu", "Yuqi Jia", "Runpeng Geng", "Jinyuan Jia", "Neil Zhenqiang Gong"]
year: 2023
venue: "arXiv:2310.12815"
source: "https://arxiv.org/abs/2310.12815"
doi: "10.48550/arxiv.2310.12815"
arxiv: "2310.12815"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-formalizing-benchmarking-prompt-injection-attacks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-formalizing-benchmarking-prompt-injection-attacks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Formalizing and Benchmarking Prompt Injection Attacks and Defenses

# Formalizing and Benchmarking Prompt Injection Attacks and Defenses

**Authors:** Yupei Liu, Yuqi Jia, [Runpeng Geng](https://md.research.elamur.ai/free/en/authors/runpeng-geng/), [Jinyuan Jia](https://md.research.elamur.ai/free/en/authors/jinyuan-jia/) and [Neil Zhenqiang Gong](https://md.research.elamur.ai/free/en/authors/neil-zhenqiang-gong/) · **Year:** 2023 · **Venue:** arXiv:2310.12815 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.12815) · [PDF](https://arxiv.org/pdf/2310.12815) · [DOI 10.48550/arxiv.2310.12815](https://doi.org/10.48550/arxiv.2310.12815) · [arXiv 2310.12815](https://arxiv.org/abs/2310.12815) · [OpenAlex](https://openalex.org/W4387839052)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/liu-2023-formalizing-benchmarking-prompt-injection-attacks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM-Integrated Applications build their prompts from an instruction and from data taken from outside sources such as resumes or webpages. Anyone who controls that data can insert text that hijacks the model. Earlier work on these prompt injection attacks consisted mostly of isolated case studies, with no formal definition and no systematic measurement. The authors define the attack formally: an adversary corrupts the data of a target task so that the application performs an injected task of the attacker's choosing instead. They then propose a generic framework in which each attack is a strategy for assembling compromised data from three ingredients: the original data, the injected instruction, and the injected data. Prior attacks fit into this framework as special cases, and merging their tricks produces a new combined attack. The authors benchmark 5 attacks and 10 defenses across 10 LLMs and 7 tasks. The combined attack is the strongest and works reliably, while none of the defenses offers adequate protection. The released platform gives future defenses a baseline set of attacks to evaluate against.

## Key points
- Benchmark scale: 5 attacks, 10 defenses (prevention- and detection-based), 10 LLMs, 7 tasks. The code is public as Open-Prompt-Injection on GitHub.
- The framework-derived combined attack beats existing attacks and stays effective across many target/injected task pairs.
- Prevention defenses rewrite the prompt or preprocess the data, for example with delimiters, instructional prevention, or sandwich prevention. They block attacks only partly and/or reduce accuracy on clean inputs.
- Known-answer detection catches compromised data in some settings but misses much of it elsewhere. The other detectors either miss many attacks or wrongly flag clean data.
- Threat model: the attacker controls only the data, not the instruction prompt or the LLM, and does not know the application's internals. Open problems include optimization-based attacks, recovering clean data after detection, and fine-tuning defenses such as Jatmo, which were not evaluated.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Duke University](https://md.research.elamur.ai/free/en/universities/duke-university/) — University, Durham, United States · 37 works · 15,861 citations
- [Pennsylvania State University](https://md.research.elamur.ai/free/en/universities/pennsylvania-state-university/) — University, State College, United States · 67 works · 5,865 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia and Neil Zhenqiang Gong. 2023. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. arXiv:2310.12815.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2023formalizing,
  title = {{Formalizing and Benchmarking Prompt Injection Attacks and Defenses}},
  author = {Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia and Neil Zhenqiang Gong},
  year = {2023},
  doi = {10.48550/arxiv.2310.12815},
  eprint = {2310.12815},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.12815}
}
```

- [← previous CoLT5: Faster Long-Range Transformers with Conditional Computation](https://md.research.elamur.ai/free/en/papers/ainslie-2023-colt5-faster-long-range-transformers/)
- [next → Hate Speech Classifiers Learn Normative Social Stereotypes](https://md.research.elamur.ai/free/en/papers/davani-2023-hate-speech-classifiers-learn-normative/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] Bing Search. [https://www.bing.com/](https://www.bing.com/), 2023. [↗](https://www.bing.com/)
2. [2] ChatGPT Plugins. [https://openai.com/blog/](https://openai.com/blog/) chatgpt-plugins, 2023. [↗](https://openai.com/blog/)
3. [15] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In NeurIPS, 2020. [https://gptstore.ai/plugins/](https://gptstore.ai/plugins/) [3] ChatWithPDF. chatwithpdf-sdan-io, 2023. [↗](https://gptstore.ai/plugins/)
4. [4] Instruction defense. [https://learnprompting.org/docs/prompt_hacking/defensive_measures/](https://learnprompting.org/docs/prompt_hacking/defensive_measures/) instruction, 2023. [↗](https://learnprompting.org/docs/prompt_hacking/defensive_measures/)
5. [16] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In USENIX Security, 2021.
6. [5] Introducing ChatGPT. [https://openai.com/blog/](https://openai.com/blog/) chatgpt, 2023. https://huggingface.co/ [6] llma2-13b-chat-url. meta-llama/Llama-2-7b, 2023. [↗](https://openai.com/blog/)
7. [7] llma2-7b-chat-url. [https://huggingface.co/](https://huggingface.co/) meta-llama/Llama-2-13b-chat-hf, 2023. [↗](https://huggingface.co/)
8. [17] Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries. arXiv, 2024. [find](https://scholar.google.com/scholar?q=%5B17%5D+Sizhe+Chen%2C+Julien+Piet%2C+Chawin+Sitawarin%2C+and+David+Wagner%2E+Struq%3A+Defending+against+prompt+injection+with+structured+queries%2E+arXiv%2C+2024%2E)
9. [8] Random sequence enclosure. [https://learnprompting.org/docs/prompt_hacking/](https://learnprompting.org/docs/prompt_hacking/) defensive_measures/random_sequence, 2023. [↗](https://learnprompting.org/docs/prompt_hacking/)
10. [18] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. [find](https://scholar.google.com/scholar?q=%5B18%5D+Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+Vicuna%3A+An+open%2Dsource+chat)
11. [9] Sandwitch defense. [https://learnprompting.org/docs/prompt_hacking/defensive_measures/](https://learnprompting.org/docs/prompt_hacking/defensive_measures/) sandwich_defense, 2023. [↗](https://learnprompting.org/docs/prompt_hacking/defensive_measures/)
12. [19] Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. [Automated hate speech detection and the problem of offensive language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/). In ICWSM, 2017.
13. [10] Tiago A. Almeida, Jose Maria Gomez Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: New collection and results. In DOCENG, 2011. [find](https://scholar.google.com/scholar?q=%5B10%5D+Tiago+A%2E+Almeida%2C+Jose+Maria+Gomez+Hidalgo%2C+and+Akebo+Yamakami%2E+Contributions+to+the+study+of+sms+spam+filtering%3A+New+collection+and+results%2E+In+DOCENG%2C+2011%2E)
14. [20] William B. Dolan and Chris Brockett. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In IWP, 2005.
15. [11] Gabriel Alon and Michael Kamfonas. [Detecting language model attacks with perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/). arXiv, 2023.
16. [21] Hugo Touvron et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv, 2023.
17. [34] OWASP. OWASP Top 10 for Large Language Model Applications. [https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_1.pdf](https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_1.pdf), 2023. [↗](https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_1.pdf)
18. [22] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. arXiv, 2023. — [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/)
19. [35] Fábio Perez and Ian Ribeiro. [Ignore previous prompt: Attack techniques for language models](https://md.research.elamur.ai/free/en/papers/perez-2022-ignore-previous-prompt-attack-techniques/). In NeurIPS ML Safety Workshop, 2022.
20. [23] Rich Harang. Securing LLM Systems Against Prompt Injection. [https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection](https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection), 2023. [↗](https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection)
21. [36] Sundar Pichai. An important next step on our AI journey. [https://blog.google/technology/ai/](https://blog.google/technology/ai/) bard-google-ai-search-updates/, 2023. [↗](https://blog.google/technology/ai/)
22. [24] Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. Predicting grammaticality on an ordinal scale. In ACL, 2014. [find](https://scholar.google.com/scholar?q=%5B24%5D+Michael+Heilman%2C+Aoife+Cahill%2C+Nitin+Madnani%2C+Melissa+Lopez%2C+Matthew+Mulholland%2C+and+Joel+Tetreault%2E+Predicting+grammaticality+on+an+ordinal+scale%2E+In+ACL%2C+2014%2E)
23. [37] Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. Jatmo: Prompt injection defense by task-specific finetuning. arXiv, 2024. [find](https://scholar.google.com/scholar?q=%5B37%5D+Julien+Piet%2C+Maha+Alrashed%2C+Chawin+Sitawarin%2C+Sizhe+Chen%2C+Zeming+Wei%2C+Elizabeth+Sun%2C+Basel+Alomair%2C+and+David+Wagner%2E+Jatmo%3A+Prompt+injection+defense+by+task%2Dspecific+finetuning%2E+arXiv%2C+2024%2E)
24. [25] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. [Baseline defenses for adversarial attacks against aligned language models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/). arXiv, 2023.
25. [38] Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. [BPE-dropout: Simple and effective subword regularization](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/). In ACL, 2020.
26. [39] Alexander M. Rush, Sumit Chopra, and Jason Weston. [A neural attention model for abstractive sentence summarization](https://md.research.elamur.ai/free/en/papers/rush-2015-neural-attention-model-abstractive-sentence/). EMNLP, 2015.
27. [26] Chin-Yew Lin. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, 2004.
28. [40] Jose Selvi. Exploring Prompt Injection Attacks. [https://research.nccgroup.com/2022/12/05/](https://research.nccgroup.com/2022/12/05/) exploring-prompt-injection-attacks/, 2022. [↗](https://research.nccgroup.com/2022/12/05/)
29. [27] Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. [Prompt injection attack against llm-integrated applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/). arXiv, 2023.
30. [41] Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. In CCS, 2021. [find](https://scholar.google.com/scholar?q=%5B41%5D+Lujia+Shen%2C+Shouling+Ji%2C+Xuhong+Zhang%2C+Jinfeng+Li%2C+Jing+Chen%2C+Jie+Shi%2C+Chengfang+Fang%2C+Jianwei+Yin%2C+and+Ting+Wang%2E+Backdoor+pre%2Dtrained+models+can+transfer+to+all%2E+In+CCS%2C+2021%2E)
31. [28] James Manyika. An overview of Bard: an early experiment with generative AI. [https://ai.google/static/documents/google-about-bard.pdf](https://ai.google/static/documents/google-about-bard.pdf), 2023. [↗](https://ai.google/static/documents/google-about-bard.pdf)
32. [42] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In EMNLP, 2013.
33. [29] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. [Membership inference attacks against language models via neighbourhood comparison](https://md.research.elamur.ai/free/en/papers/mattern-2023-membership-inference-attacks-against-language/). In ACL Findings, 2023.
34. [43] R Gorman Stuart Armstrong. Using GPT-Eliezer against ChatGPT Jailbreaking. [https://www.alignmentforum.org/posts/pNcFYZnPdXy](https://www.alignmentforum.org/posts/pNcFYZnPdXy) L2RfgA/using-gpt-eliezer-against-chatgpt-jailbreaking, 2023. [↗](https://www.alignmentforum.org/posts/pNcFYZnPdXy)
35. [30] Alexandra Mendes. Ultimate ChatGPT prompt engineering guide for general users and developers. [https://www.imaginarycloud.com/blog/](https://www.imaginarycloud.com/blog/) chatgpt-prompt-engineering, 2023. [↗](https://www.imaginarycloud.com/blog/)
36. Yohei’s blog post. [31] Yohei Nakajima. //twitter.com/yoheinakajima/status/ 1582844144640471040, 2022. [find](https://scholar.google.com/scholar?q=Yohei%E2%80%99s+blog+post%2E+%5B31%5D+Yohei+Nakajima%2E+%2F%2Ftwitter%2Ecom%2Fyoheinakajima%2Fstatus%2F+1582844144640471040%2C+2022%2E)
37. [44] Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. [Ul2: Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). In ICLR, 2023. https:
38. [32] Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. Jfleg: A fluency corpus and benchmark for grammatical error correction. In EACL, 2017. [find](https://scholar.google.com/scholar?q=%5B32%5D+Courtney+Napoles%2C+Keisuke+Sakaguchi%2C+and+Joel+Tetreault%2E+Jfleg%3A+A+fluency+corpus+and+benchmark+for+grammatical+error+correction%2E+In+EACL%2C+2017%2E)
39. [45] InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM), 2023. [↗](https://github.com/InternLM/InternLM)
40. [33] OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, 2023. *Table 10: ASVs of different attacks averaged over the 7 × 7 target/injected task combinations. The LLM is PaLM 2.* [46] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In ICML, 2023.
41. [47] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR, 2019.
42. Escape Characters Context Ignoring Fake Completion Combined Attack 0.62 0.64 0.65 0.66 0.71 [find](https://scholar.google.com/scholar?q=Escape+Characters+Context+Ignoring+Fake+Completion+Combined+Attack+0%2E62+0%2E64+0%2E65+0%2E66+0%2E71)

*Updated 2 November 2026*