---
title: "The Instruction Hierarchy — paper by Wallace et al. (2024)"
description: "They propose an explicit instruction hierarchy with system messages on top, user messages next… 33 references, cited by 73 papers here."
authors: ["Eric Wallace", "Kai Xiao", "Reimar Leike", "Lilian Weng", "Johannes Heidecke", "Alex Beutel"]
year: 2024
venue: "arXiv:2404.13208"
source: "https://arxiv.org/abs/2404.13208"
doi: "10.48550/arxiv.2404.13208"
arxiv: "2404.13208"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wallace-2024-instruction-hierarchy-training-llms-prioritize/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wallace-2024-instruction-hierarchy-training-llms-prioritize/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

# The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

**Authors:** [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), Kai Xiao, Reimar Leike, [Lilian Weng](https://md.research.elamur.ai/free/en/authors/lilian-weng/), [Johannes Heidecke](https://md.research.elamur.ai/free/en/authors/johannes-heidecke/) and [Alex Beutel](https://md.research.elamur.ai/free/en/authors/alex-beutel/) · **Year:** 2024 · **Venue:** arXiv:2404.13208 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.13208) · [PDF](https://arxiv.org/pdf/2404.13208) · [DOI 10.48550/arxiv.2404.13208](https://doi.org/10.48550/arxiv.2404.13208) · [arXiv 2404.13208](https://arxiv.org/abs/2404.13208) · [OpenAlex](https://openalex.org/W4395064792)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/wallace-2024-instruction-hierarchy-training-llms-prioritize/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLM-based applications can be hijacked by prompt injections, jailbreaks and system prompt extraction. The authors trace much of this weakness to one cause. Models give developer instructions, user requests and third-party content (such as web results or tool returns) roughly equal weight. They propose an explicit instruction hierarchy with system messages on top, user messages next, and tool outputs lowest. A model following it carries out lower-level instructions that agree with higher-level goals. It disregards conflicting ones, or declines when it cannot proceed.

To teach this behaviour, they build training data automatically. Benign compound requests are split into fragments spread across hierarchy levels, and the model learns to reproduce the original answer. Adversarial examples come from red-teaming LLMs, and context distillation trains the model to respond as if the injected instructions were never there. GPT-3.5 Turbo fine-tuned on this data becomes far more robust, including against attack types absent from training. General abilities are largely preserved, though over-refusal increases somewhat. This makes the method relevant for safer agent-style deployments.

## Key points
- Privilege order: System Message > User Message > Tool Outputs. Lower-level instructions are labelled either aligned (follow them) or misaligned (ignore or refuse them).
- Training combined synthetic data and context distillation with generic instruction-following data. GPT-3.5 Turbo was trained using supervised fine-tuning and RLHF.
- Protection against system prompt extraction improved by 63%. Jailbreak robustness rose by over 30%, even though jailbreaks were not directly modelled in data generation.
- Over-refusal benchmarks showed near-baseline compliance on most tasks. Two sets were exceptions: the one posing probing questions about the system message, and the Jailbreakchat set made up of allowed prompts. Both were adversarially constructed from harmless inputs that look like attacks.
- Planned future work includes multimodal injections (images, audio), architectural changes such as level-specific message embeddings, and stronger adversarial training. The authors acknowledge the models likely remain vulnerable to powerful attacks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke and Alex Beutel. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wallace2024instruction,
  title = {{The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions}},
  author = {Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke and Alex Beutel},
  year = {2024},
  doi = {10.48550/arxiv.2404.13208},
  eprint = {2404.13208},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.13208}
}
```

- [← previous Mitigating the Alignment Tax of RLHF](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/)
- [next → Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://md.research.elamur.ai/free/en/papers/zhou-2024-transfusion-predict-next-token-diffuse/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
2. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. StruQ: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363, 2024. — [arXiv:2402.06363](https://arxiv.org/abs/2402.06363) [find](https://scholar.google.com/scholar?q=Sizhe+Chen%2C+Julien+Piet%2C+Chawin+Sitawarin%2C+and+David+Wagner%2E+StruQ%3A+Defending+against+prompt+injection+with+structured+queries%2E+arXiv+preprint+arXiv%3A2402%2E06363%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E06363%5D%28https%3A%2F%2Farxi)
3. Fernando J Corbató and Victor A Vyssotsky. Introduction and overview of the Multics system. In November 30–December 1, 1965, Fall Joint Computer Conference, Part I, 1965. [find](https://scholar.google.com/scholar?q=Fernando+J+Corbato%CC%81+and+Victor+A+Vyssotsky%2E+Introduction+and+overview+of+the+Multics+system%2E+In+November+30%E2%80%93December+1%2C+1965%2C+Fall+Joint+Computer+Conference%2C+Part+I%2C+1965%2E)
4. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. [Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/). arXiv preprint arXiv:2209.07858, 2022.
5. Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing LLMs to do and reveal (almost) anything. arXiv preprint arXiv:2402.14020, 2024. — [arXiv:2402.14020](https://arxiv.org/abs/2402.14020) [find](https://scholar.google.com/scholar?q=Jonas+Geiping%2C+Alex+Stein%2C+Manli+Shu%2C+Khalid+Saifullah%2C+Yuxin+Wen%2C+and+Tom+Goldstein%2E+Coercing+LLMs+to+do+and+reveal+%28almost%29+anything%2E+arXiv+preprint+arXiv%3A2402%2E14020%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E14020%5D%28http)
6. Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
7. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In ACM Workshop on Artificial Intelligence and Security, 2023. — [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/)
8. Lakera AI. Gandalf game—Level 4 adventure, 2023. URL [https://huggingface.co/datasets/Lakera/gandalf](https://huggingface.co/datasets/Lakera/gandalf) summarization. [↗](https://huggingface.co/datasets/Lakera/gandalf)
9. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. [Prompt injection attack against LLM-integrated applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/). arXiv preprint arXiv:2306.05499, 2023.
10. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332, 2021.
11. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
12. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In Proceedings of Advances in Neural Information Processing Syste…
13. Aaron Parisi, Yao Zhao, and Noah Fiedel. [TALM: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/). arXiv preprint arXiv:2205.12255, 2022.
14. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). In EMNLP, 2022.
15. Fábio Perez and Ian Ribeiro. [Ignore previous prompt: Attack techniques for language models](https://md.research.elamur.ai/free/en/papers/perez-2022-ignore-previous-prompt-attack-techniques/). In NeurIPS ML Safety Workshop, 2022.
16. Dennis M. Ritchie and Ken Thompson. The UNIX time-sharing system. Communications of the ACM, 1974. [find](https://scholar.google.com/scholar?q=Dennis+M%2E+Ritchie+and+Ken+Thompson%2E+The+UNIX+time%2Dsharing+system%2E+Communications+of+the+ACM%2C+1974%2E)
17. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in Neural Information Processing Systems, 36, 2024.
18. Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan Boyd-Graber. Ignore this title and HackAPrompt: Exposing systemic vulnerabilities of llms through a global scale prompt hacking competi… [find](https://scholar.google.com/scholar?q=Sander+Schulhoff%2C+Jeremy+Pinto%2C+Anaum+Khan%2C+Louis%2DFranc%CC%A7ois+Bouchard%2C+Chenglei+Si%2C+Svetlina+Anati%2C+Valen+Tagliabue%2C+Anson+Liu+Kost%2C+Christopher+Carnahan%2C+and+Jordan+Boyd%2DGraber%2E+Ignore+this+title+and+)
19. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. [HuggingGPT: Solving AI tasks with ChatGPT and its friends in hugging face](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/). NeurIPS, 2024.
20. Charlie Snell, Dan Klein, and Ruiqi Zhong. [Learning by distilling context](https://md.research.elamur.ai/free/en/papers/snell-2022-learning-distilling-context/). arXiv preprint arXiv:2209.15189, 2022.
21. Zhendong Su and Gary Wassermann. The essence of command injection attacks in web applications. In ACM-SIGACT Symposium on Principles of Programming Languages, 2006. [find](https://scholar.google.com/scholar?q=Zhendong+Su+and+Gary+Wassermann%2E+The+essence+of+command+injection+attacks+in+web+applications%2E+In+ACM%2DSIGACT+Symposium+on+Principles+of+Programming+Languages%2C+2006%2E)
22. Stephen Thomas, Laurie Williams, and Tao Xie. On automated prepared statement generation to remove SQL injection vulnerabilities. Information and Software Technology, 2009. [find](https://scholar.google.com/scholar?q=Stephen+Thomas%2C+Laurie+Williams%2C+and+Tao+Xie%2E+On+automated+prepared+statement+generation+to+remove+SQL+injection+vulnerabilities%2E+Information+and+Software+Technology%2C+2009%2E)
23. Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. Tensor Trust: Interpretable prompt injection attacks from an online game. In ICLR, 2024. [find](https://scholar.google.com/scholar?q=Sam+Toyer%2C+Olivia+Watkins%2C+Ethan+Adrian+Mendes%2C+Justin+Svegliato%2C+Luke+Bailey%2C+Tiffany+Wang%2C+Isaac+Ong%2C+Karim+Elmaaroufi%2C+Pieter+Abbeel%2C+Trevor+Darrell%2C+et+al%2E+Tensor+Trust%3A+Interpretable+prompt+injec)
24. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+Nikhil+Kandpal%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+Universal+adversarial+triggers+for+attacking+and+analyzing+NLP%2E+In+EMNLP%2C+2019%2E)
25. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. [Jailbroken: How does LLM safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) In NeurIPS, 2023.
26. Lilian Weng. LLM-powered autonomous agents. lilianweng.github.io, Jun 2023. URL [https://lilianweng.github.io/posts/2023-06-23-agent/](https://lilianweng.github.io/posts/2023-06-23-agent/). [↗](https://lilianweng.github.io/posts/2023-06-23-agent/)
27. Simon Willison. Prompt injection attacks against GPT-3, 2022. URL [https://simonwillison.net/2022/Sep/12/prompt-injection/](https://simonwillison.net/2022/Sep/12/prompt-injection/). [↗](https://simonwillison.net/2022/Sep/12/prompt-injection/)
28. Simon Willison. Multi-modal prompt injection image attacks against GPT-4V, 2023. URL [https://simonwillison.net/2023/Oct/14/multi-modal-prompt-injection/](https://simonwillison.net/2023/Oct/14/multi-modal-prompt-injection/). [↗](https://simonwillison.net/2023/Oct/14/multi-modal-prompt-injection/)
29. Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197, 2023. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) [find](https://scholar.google.com/scholar?q=Jingwei+Yi%2C+Yueqi+Xie%2C+Bin+Zhu%2C+Keegan+Hines%2C+Emre+Kiciman%2C+Guangzhong+Sun%2C+Xing+Xie%2C+and+Fangzhao+Wu%2E+Benchmarking+and+defending+against+indirect+prompt+injection+attacks+on+large+language+models%2E+ar)
30. Yiming Zhang and Daphne Ippolito. Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success. arXiv preprint arXiv:2307.06865, 2023. — [arXiv:2307.06865](https://arxiv.org/abs/2307.06865) [find](https://scholar.google.com/scholar?q=Yiming+Zhang+and+Daphne+Ippolito%2E+Prompts+should+not+be+seen+as+secrets%3A+Systematically+measuring+prompt+extraction+attack+success%2E+arXiv+preprint+arXiv%3A2307%2E06865%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E06865%5D%28https%3A%2F%2F)
31. Weilin Zhong, Wichers, Amwestgate, Rezos, Clow808, KristenS, Jason Li, Andrew Smith, Jmanico, Tal Mel, and kingthorin. Command injection — OWASP foundation, 2024. URL [https://owasp.org/wwwcommunity/attacks/Command](https://owasp.org/wwwcommunity/attacks/Command) Injection. [↗](https://owasp.org/wwwcommunity/attacks/Command)
32. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043, 2023.
33. Egor Zverev, Sahar Abdelnabi, Mario Fritz, and Christoph H Lampert. Can LLMs separate instructions from data? And what do we even mean by that? In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. [find](https://scholar.google.com/scholar?q=Egor+Zverev%2C+Sahar+Abdelnabi%2C+Mario+Fritz%2C+and+Christoph+H+Lampert%2E+Can+LLMs+separate+instructions+from+data%3F+And+what+do+we+even+mean+by+that%3F+In+ICLR+2024+Workshop+on+Secure+and+Trustworthy+Large+La)

*Updated 2 November 2026*