---
title: "Visual Adversarial Examples Jailbreak Aligned Large Language"
description: "To show this, they build an image perturbation for vision-language models that were tuned to refuse… 84 references, cited by 95 papers here."
authors: ["Xiangyu Qi", "Kaixuan Huang", "Ashwinee Panda", "Peter Henderson", "Mengdi Wang", "Prateek Mittal"]
year: 2023
venue: "arXiv:2306.13213"
source: "https://arxiv.org/abs/2306.13213"
doi: "10.48550/arxiv.2306.13213"
arxiv: "2306.13213"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qi-2023-visual-adversarial-examples-jailbreak-aligned/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qi-2023-visual-adversarial-examples-jailbreak-aligned/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Visual Adversarial Examples Jailbreak Aligned Large Language Models

# Visual Adversarial Examples Jailbreak Aligned Large Language Models

**Authors:** [Xiangyu Qi](https://md.research.elamur.ai/free/en/authors/xiangyu-qi/), [Kaixuan Huang](https://md.research.elamur.ai/free/en/authors/kaixuan-huang/), [Ashwinee Panda](https://md.research.elamur.ai/free/en/authors/ashwinee-panda/), [Peter Henderson](https://md.research.elamur.ai/free/en/authors/peter-henderson/), [Mengdi Wang](https://md.research.elamur.ai/free/en/authors/mengdi-wang/) and [Prateek Mittal](https://md.research.elamur.ai/free/en/authors/prateek-mittal/) · **Year:** 2023 · **Venue:** [arXiv:2306.13213](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2023](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2023/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.13213) · [PDF](https://arxiv.org/pdf/2306.13213) · [DOI 10.48550/arxiv.2306.13213](https://doi.org/10.48550/arxiv.2306.13213) · [arXiv 2306.13213](https://arxiv.org/abs/2306.13213) · [OpenAlex](https://openalex.org/W4393157467)

**Citations:** 145 citations (OpenAlex, 2026-10-14) · [95](https://md.research.elamur.ai/free/en/papers/qi-2023-visual-adversarial-examples-jailbreak-aligned/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Adding image inputs to large language models creates new security risks. The authors argue that images are continuous and high-dimensional, which makes them much easier to manipulate adversarially than discrete text. Because these models can do so many things, a successful attack can do more than cause a misclassification.

To show this, they build an image perturbation for vision-language models that were tuned to refuse harmful requests. The image is optimized to make the model more likely to reproduce a small set of toxic sentences aimed at a few identity groups. Surprisingly, one such image works as a general jailbreak. When it accompanies a text prompt, the model follows many harmful instructions, including types of harm absent from the optimization corpus.

The authors confirm the effect with manual review of varied harmful scenarios and with automated toxicity scoring. They also show that the attack partly transfers between models in a black-box setting.

The work suggests that alignment methods trained only on text, such as instruction tuning or RLHF, do not protect against attacks through other input channels. It frames adversarial robustness as an open problem for AI alignment.

## Key points
- The adversarial image was optimized on a few-shot corpus of 66 derogatory sentences. These targeted two anonymized identities (\<gender-1>, \<race-1>) and humanity as a whole, under a perturbation budget of ε = 16/255.
- Three models were attacked: MiniGPT-4 and InstructBLIP (both on Vicuna) and LLaVA (on LLaMA-2). The jailbreak effect appeared in all three, and black-box transfer between them was demonstrated.
- In a MiniGPT-4 example, a hateful request was refused 98% of the time with a clean image. With the adversarial image, the model complied 83% of the time. A request for murder instructions, which lay outside the training corpus, went from 70% refusal to 78% compliance. Each figure is based on 100 sampled outputs checked by hand.
- Automated evaluation used RealToxicityPrompts, scored with Perspective API and Detoxify. The two scorers disagreed noticeably, and Perspective flagged more outputs as toxic.
- The attack code is public on GitHub (Unispac/Visual-Adversarial-Examples-Jailbreak-Large-Language-Models). The paper also discusses possible mitigations and implications for AI policy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 113 works · 48,612 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang and Prateek Mittal. 2023. Visual Adversarial Examples Jailbreak Aligned Large Language Models. arXiv:2306.13213.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qi2023visual,
  title = {{Visual Adversarial Examples Jailbreak Aligned Large Language Models}},
  author = {Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang and Prateek Mittal},
  year = {2023},
  doi = {10.48550/arxiv.2306.13213},
  eprint = {2306.13213},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.13213}
}
```

- [← previous FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/)
- [next → Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

84 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adversa. Universal LLM jailbreak: ChatGPT, GPT-4, Bard, Bing, Anthropic, and beyond. [https://adversa.ai/blog/universal-llm-jailbreak-chatgpt-gpt-4-bard-bing-anthropic-and-beyond/](https://adversa.ai/blog/universal-llm-jailbreak-chatgpt-gpt-4-bard-bing-anthropic-and-beyond/) , 2023. 4 [↗](https://adversa.ai/blog/universal-llm-jailbreak-chatgpt-gpt-4-bard-bing-anthropic-and-beyond/)
2. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716–23736, 2022. 1, 3
3. M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang. [Generating natural language adversarial examples](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/). arXiv preprint arXiv:1804.07998, 2018. 1, 4
4. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. 1 [find](https://scholar.google.com/scholar?q=S%2E+Antol%2C+A%2E+Agrawal%2C+J%2E+Lu%2C+M%2E+Mitchell%2C+D%2E+Batra%2C+C%2E+L%2E+Zitnick%2C+and+D%2E+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+2425%E2%80%93243)
5. A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pages 274–283. PMLR, 2018. 1, 4, 9 [find](https://scholar.google.com/scholar?q=A%2E+Athalye%2C+N%2E+Carlini%2C+and+D%2E+Wagner%2E+Obfuscated+gradients+give+a+false+sense+of+security%3A+Circumventing+defenses+to+adversarial+examples%2E+In+International+conference+on+machine+learning%2C+pages+274%E2%80%932)
6. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J.… [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+A%2E+Askell%2C+A%2E+Chen%2C+N%2E+DasSarma%2C+D%2E+Drain%2C+S%2E+Fort%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+N%2E+Joseph%2C+S%2E+Kadavath%2C+J%2E+Kernion%2C+T%2E+Conerly%2C+S%2E+El%2DShowk%2C+N%2E+Elhage%2C+Z%2E+Hatfield%2DDodds%2C+D%2E)
7. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022. 4
8. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258, 2021. 3, 10
9. A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 3, 11 [find](https://scholar.google.com/scholar?q=A%2E+Brohan%2C+N%2E+Brown%2C+J%2E+Carbajal%2C+Y%2E+Chebotar%2C+X%2E+Chen%2C+K%2E+Choromanski%2C+T%2E+Ding%2C+D%2E+Driess%2C+A%2E+Dubey%2C+C%2E+Finn%2C+et+al%2E+Rt%2D2%3A+Vision%2Dlanguage%2Daction+models+transfer+web+knowledge+to+robotic+control%2E+arX)
10. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020. 1, 3
11. T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou. [Large language models as tool makers](https://md.research.elamur.ai/free/en/papers/cai-2023-large-language-models-tool-makers/). arXiv preprint arXiv:2305.17126, 2023. 11
12. Y. Cao, N. Wang, C. Xiao, D. Yang, J. Fang, R. Yang, Q. A. Chen, M. Liu, and B. Li. Invisible for both camera and lidar: Security of multi-sensor fusion based perception in autonomous driving under physical-world attacks. In 2021 IEEE Symposium on Security and Privacy (SP), pages 176–194. IEEE, 2021… [find](https://scholar.google.com/scholar?q=Y%2E+Cao%2C+N%2E+Wang%2C+C%2E+Xiao%2C+D%2E+Yang%2C+J%2E+Fang%2C+R%2E+Yang%2C+Q%2E+A%2E+Chen%2C+M%2E+Liu%2C+and+B%2E+Li%2E+Invisible+for+both+camera+and+lidar%3A+Security+of+multi%2Dsensor+fusion+based+perception+in+autonomous+driving+under+ph)
13. N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 3–14, 2017. 1, 4, 9 [find](https://scholar.google.com/scholar?q=N%2E+Carlini+and+D%2E+Wagner%2E+Adversarial+examples+are+not+easily+detected%3A+Bypassing+ten+detection+methods%2E+In+Proceedings+of+the+10th+ACM+workshop+on+artificial+intelligence+and+security%2C+pages+3%E2%80%9314%2C+20)
14. N. Carlini and D. Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE security and privacy workshops (SPW), pages 1–7. IEEE, 2018. 3 [find](https://scholar.google.com/scholar?q=N%2E+Carlini+and+D%2E+Wagner%2E+Audio+adversarial+examples%3A+Targeted+attacks+on+speech%2Dto%2Dtext%2E+In+2018+IEEE+security+and+privacy+workshops+%28SPW%29%2C+pages+1%E2%80%937%2E+IEEE%2C+2018%2E+3)
15. N. Carlini, F. Tramer, J. Z. Kolter, et al. arXiv:2206.10550, 2022. 10 (certified!!) adversarial robustness for free! — [arXiv:2206.10550](https://arxiv.org/abs/2206.10550) [find](https://scholar.google.com/scholar?q=N%2E+Carlini%2C+F%2E+Tramer%2C+J%2E+Z%2E+Kolter%2C+et+al%2E+arXiv%3A2206%2E10550%2C+2022%2E+10+%28certified%21%21%29+adversarial+robustness+for+free%21+%E2%80%94+%5BarXiv%3A2206%2E10550%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E10550%29)
16. N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, A. Awadalla, P. W. Koh, D. Ippolito, K. Lee, F. Tramer, et al. [Are aligned neural networks adversarially aligned?](https://md.research.elamur.ai/free/en/papers/carlini-2023-aligned-neural-networks-adversarially-aligned/) arXiv preprint arXiv:2306.15447, 2023. 4
17. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/) . 3, 6 [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
18. J. Christian. Amazing “jailbreak” bypasses ChatGPT’s ethics safeguards. [https://futurism.com/amazing](https://futurism.com/amazing) -jailbreak-chatgpt , 2023. 4 [↗](https://futurism.com/amazing)
19. J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310–1320. PMLR, 2019. 10 [find](https://scholar.google.com/scholar?q=J%2E+Cohen%2C+E%2E+Rosenfeld%2C+and+Z%2E+Kolter%2E+Certified+adversarial+robustness+via+randomized+smoothing%2E+In+international+conference+on+machine+learning%2C+pages+1310%E2%80%931320%2E+PMLR%2C+2019%2E+10)
20. F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020. 10 arXiv preprint — [arXiv:2010.09670](https://arxiv.org/abs/2010.09670) [find](https://scholar.google.com/scholar?q=F%2E+Croce%2C+M%2E+Andriushchenko%2C+V%2E+Sehwag%2C+E%2E+Debenedetti%2C+N%2E+Flammarion%2C+M%2E+Chiang%2C+P%2E+Mittal%2C+and+M%2E+Hein%2E+Robustbench%3A+a+standardized+adversarial+robustness+benchmark%2E+arXiv+preprint+arXiv%3A2010%2E09670%2C)
21. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023. 2, 3, 6, 16, 17
22. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 11 [find](https://scholar.google.com/scholar?q=D%2E+Driess%2C+F%2E+Xia%2C+M%2E+S%2E+Sajjadi%2C+C%2E+Lynch%2C+A%2E+Chowdhery%2C+B%2E+Ichter%2C+A%2E+Wahid%2C+J%2E+Tompson%2C+Q%2E+Vuong%2C+T%2E+Yu%2C+et+al%2E+Palm%2De%3A+An+embodied+multimodal+language+model%2E+arXiv+preprint+arXiv%3A2303%2E03378%2C+2023%2E)
23. J. Ebrahimi, A. Rao, D. Lowd, and D. Dou. [Hotflip: White-box adversarial examples for text classification](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/). arXiv preprint arXiv:1712.06751, 2017. 6
24. Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023. 6 [find](https://scholar.google.com/scholar?q=Y%2E+Fang%2C+W%2E+Wang%2C+B%2E+Xie%2C+Q%2E+Sun%2C+L%2E+Wu%2C+X%2E+Wang%2C+T%2E+Huang%2C+X%2E+Wang%2C+and+Y%2E+Cao%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+)
25. D. Ganguli, D. Hernandez, L. Lovitt, A. Askell, Y. Bai, A. Chen, T. Conerly, N. Dassarma, D. Drain, N. Elhage, et al. [Predictability and surprise in large generative models](https://md.research.elamur.ai/free/en/papers/ganguli-2022-predictability-surprise-large-generative-models/). In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022. 11
26. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. [Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/). arXiv preprint arXiv:2209.07858, 2022. 4, 7, 20
27. Y. Gao, I. Shumailov, K. Fawaz, and N. Papernot. On the limitations of stochastic pre-processing defenses. arXiv preprint arXiv:2206.09491, 2022. 10 — [arXiv:2206.09491](https://arxiv.org/abs/2206.09491) [find](https://scholar.google.com/scholar?q=Y%2E+Gao%2C+I%2E+Shumailov%2C+K%2E+Fawaz%2C+and+N%2E+Papernot%2E+On+the+limitations+of+stochastic+pre%2Dprocessing+defenses%2E+arXiv+preprint+arXiv%3A2206%2E09491%2C+2022%2E+10+%E2%80%94+%5BarXiv%3A2206%2E09491%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E094)
28. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. [Realtoxicityprompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). arXiv preprint arXiv:2009.11462, 2020. 2, 3, 7
29. R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. [Imagebind: One embedding space to bind them all](https://md.research.elamur.ai/free/en/papers/girdhar-2023-imagebind-one-embedding-space-bind/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 3
30. I. J. Goodfellow, J. Shlens, and C. Szegedy. [Explaining and harnessing adversarial examples](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/). arXiv preprint arXiv:1412.6572, 2014. 4
31. L. Hanu and Unitary team. Detoxify. Github. [https://github.com/unitaryai/detoxify](https://github.com/unitaryai/detoxify), 2020. 7 [↗](https://github.com/unitaryai/detoxify)
32. T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar. [Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3309–332…
33. A. Helbling, M. Phute, M. Hull, and D. H. Chau. [Llm self defense: By self examination, llms know they are being tricked](https://md.research.elamur.ai/free/en/papers/phute-2023-llm-self-defense-self-examination/), 2023. 10
34. P. Henderson, M. Krass, L. Zheng, N. Guha, C. D. Manning, D. Jurafsky, and D. Ho. [Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset](https://md.research.elamur.ai/free/en/papers/henderson-2022-pile-law-learning-responsible-data/). Advances in Neural Information Processing Systems, 35:29217–29234, 2022. 20
35. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 10 [find](https://scholar.google.com/scholar?q=J%2E+Ho%2C+A%2E+Jain%2C+and+P%2E+Abbeel%2E+Denoising+diffusion+probabilistic+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A6840%E2%80%936851%2C+2020%2E+10)
36. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). arXiv preprint arXiv:1904.09751, 2019. 6
37. S. Huang, Z. Jiang, H. Dong, Y. Qiao, P. Gao, and H. Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176, 2023. 11 — [arXiv:2305.11176](https://arxiv.org/abs/2305.11176) [find](https://scholar.google.com/scholar?q=S%2E+Huang%2C+Z%2E+Jiang%2C+H%2E+Dong%2C+Y%2E+Qiao%2C+P%2E+Gao%2C+and+H%2E+Li%2E+Instruct2act%3A+Mapping+multi%2Dmodality+instructions+to+robotic+actions+with+large+language+model%2E+arXiv+preprint+arXiv%3A2305%2E11176%2C+2023%2E+11+%E2%80%94+%5Bar)
38. A. Ilyas, L. Engstrom, A. Athalye, and J. Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137–2146. PMLR, 2018. 11 [find](https://scholar.google.com/scholar?q=A%2E+Ilyas%2C+L%2E+Engstrom%2C+A%2E+Athalye%2C+and+J%2E+Lin%2E+Black%2Dbox+adversarial+attacks+with+limited+queries+and+information%2E+In+International+conference+on+machine+learning%2C+pages+2137%E2%80%932146%2E+PMLR%2C+2018%2E+11)
39. E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt. [Automatically auditing large language models via discrete optimization](https://md.research.elamur.ai/free/en/papers/jones-2023-automatically-auditing-large-language-models/). arXiv preprint arXiv:2303.04381, 2023. 1, 4
40. Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. [Alignment of language agents](https://md.research.elamur.ai/free/en/papers/kenton-2021-alignment-language-agents/). arXiv preprint arXiv:2103.14659, 2021. 3
41. M. King. Meet dan — the ‘jailbreak’ version of chatgpt and how to use it — ai unchained and unfiltered. [https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-u](https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-u) se-it-ai-unchained-and-unfiltered-f91bfa679024 , 2023. 4 [↗](https://medium.com/@neonforge/meet-dan-the-jailbreak-version-of-chatgpt-and-how-to-u)
42. LAION. Open-assistant. [https://github.com/LAION-AI/Open-Assistant](https://github.com/LAION-AI/Open-Assistant) , 2023. 20 [↗](https://github.com/LAION-AI/Open-Assistant)
43. LangChain. Langchain as an aiplugin. [https://github.com/langchain-ai/langchain-aiplugin](https://github.com/langchain-ai/langchain-aiplugin) , 2023. 11 [↗](https://github.com/langchain-ai/langchain-aiplugin)
44. B. Lester, R. Al-Rfou, and N. Constant. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). arXiv preprint arXiv:2104.08691, 2021. 5
45. L. Li, T. Xie, and B. Li. Sok: Certified robustness for deep neural networks. In 44th IEEE Symposium on Security and Privacy, SP 2023, San Francisco, CA, USA, 22-26 May 2023. IEEE, 2023. 10 [find](https://scholar.google.com/scholar?q=L%2E+Li%2C+T%2E+Xie%2C+and+B%2E+Li%2E+Sok%3A+Certified+robustness+for+deep+neural+networks%2E+In+44th+IEEE+Symposium+on+Security+and+Privacy%2C+SP+2023%2C+San+Francisco%2C+CA%2C+USA%2C+22%2D26+May+2023%2E+IEEE%2C+2023%2E+10)
46. F. Liu, K.-S. Liu, C. Chang, and Y. Wang. Research on the technology of ios jailbreak. In 2016 Sixth International Conference on Instrumentation & Measurement, Computer, Communication and Control (IMCCC), pages 644–647. IEEE, 2016. 4 [find](https://scholar.google.com/scholar?q=F%2E+Liu%2C+K%2E%2DS%2E+Liu%2C+C%2E+Chang%2C+and+Y%2E+Wang%2E+Research+on+the+technology+of+ios+jailbreak%2E+In+2016+Sixth+International+Conference+on+Instrumentation+%26+Measurement%2C+Computer%2C+Communication+and+Control+%28IMC)
47. H. Liu, C. Li, Q. Wu, and Y. J. Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/), 2023. 2, 3, 6, 16, 17
48. Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu. [Jailbreaking chatgpt via prompt engineering: An empirical study](https://md.research.elamur.ai/free/en/papers/liu-2023-jailbreaking-chatgpt-prompt-engineering-empirical/). arXiv preprint arXiv:2305.13860, 2023. 2, 4
49. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. [Towards deep learning models resistant to adversarial attacks](https://md.research.elamur.ai/free/en/papers/madry-2017-towards-deep-learning-models-resistant/). arXiv preprint arXiv:1706.06083, 2017. 1, 6, 10
50. N. Mehrabi, A. Beirami, F. Morstatter, and A. Galstyan. Robust conversational agents against imperceptible toxicity triggers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2831–2847, Seattle, U… [find](https://scholar.google.com/scholar?q=N%2E+Mehrabi%2C+A%2E+Beirami%2C+F%2E+Morstatter%2C+and+A%2E+Galstyan%2E+Robust+conversational+agents+against+imperceptible+toxicity+triggers%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the)
51. Microsoft. Introduction to red teaming large language models (llms). [https://learn.microsoft.com/en](https://learn.microsoft.com/en) -us/azure/ai-services/openai/concepts/red-teaming , 2023. 4 [↗](https://learn.microsoft.com/en)
52. Z. Mowshowitz. Jailbreaking chatgpt on release day. [https://www.lesswrong.com/posts/RYcoJdvmoBbi](https://www.lesswrong.com/posts/RYcoJdvmoBbi) 5Nax7/jailbreaking-chatgpt-on-release-day , 2022. 4 [↗](https://www.lesswrong.com/posts/RYcoJdvmoBbi)
53. W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460, 2022. 10 — [arXiv:2205.07460](https://arxiv.org/abs/2205.07460) [find](https://scholar.google.com/scholar?q=W%2E+Nie%2C+B%2E+Guo%2C+Y%2E+Huang%2C+C%2E+Xiao%2C+A%2E+Vahdat%2C+and+A%2E+Anandkumar%2E+Diffusion+models+for+adversarial+purification%2E+arXiv+preprint+arXiv%3A2205%2E07460%2C+2022%2E+10+%E2%80%94+%5BarXiv%3A2205%2E07460%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F220)
54. OpenAI. Introducing ChatGPT. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt) , 2022. 1, 6 [↗](https://openai.com/blog/chatgpt)
55. OpenAI. Moderation api. [https://platform.openai.com/docs/guides/moderation](https://platform.openai.com/docs/guides/moderation) , 2023. 20 [↗](https://platform.openai.com/docs/guides/moderation)
56. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023. 1, 3, 4, 10
57. OpenAI. Forecasting potential misuses of language models for disinformation campaigns and how to reduce risk. [https://openai.com/research/forecasting-misuse](https://openai.com/research/forecasting-misuse) , 2023. [Online; accessed 4-Apr-2023]. 2 [↗](https://openai.com/research/forecasting-misuse)
58. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022. 2, 3
59. N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506–519, 2017. 11 [find](https://scholar.google.com/scholar?q=N%2E+Papernot%2C+P%2E+McDaniel%2C+I%2E+Goodfellow%2C+S%2E+Jha%2C+Z%2E+B%2E+Celik%2C+and+A%2E+Swami%2E+Practical+black%2Dbox+attacks+against+machine+learning%2E+In+Proceedings+of+the+2017+ACM+on+Asia+conference+on+computer+and+comm)
60. S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). arXiv preprint arXiv:2305.15334, 2023. 3, 11
61. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). arXiv preprint arXiv:2202.03286, 2022. 4, 11
62. S. Pichai. An important next step on our ai journey. [https://blog.google/technology/ai/bard-googl](https://blog.google/technology/ai/bard-googl) e-ai-search-updates/ , 2023. 3 [↗](https://blog.google/technology/ai/bard-googl)
63. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR, 2021. 6
64. A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury. Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks. arXiv preprint arXiv:2305.14965, 2023. 4 — [arXiv:2305.14965](https://arxiv.org/abs/2305.14965) [find](https://scholar.google.com/scholar?q=A%2E+Rao%2C+S%2E+Vashistha%2C+A%2E+Naik%2C+S%2E+Aditya%2C+and+M%2E+Choudhury%2E+Tricking+llms+into+disobedience%3A+Understanding%2C+analyzing%2C+and+preventing+jailbreaks%2E+arXiv+preprint+arXiv%3A2305%2E14965%2C+2023%2E+4+%E2%80%94+%5BarXiv%3A2305)
65. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. [High-resolution image synthesis with latent diffusion models](https://md.research.elamur.ai/free/en/papers/rombach-2021-high-resolution-image-synthesis-latent/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 10
66. T. Schick, S. Udupa, and H. Schütze. [Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/). Transactions of the Association for Computational Linguistics, 9:1408–1424, 2021. 7
67. ShareGPT.com. Sharegpt: Share your wildest chatgpt conversations with one click. [https://sharegpt.com/](https://sharegpt.com/) , 2023. 6 [↗](https://sharegpt.com/)
68. T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh. [Autoprompt: Eliciting knowledge from language models with automatically generated prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). arXiv preprint arXiv:2010.15980, 2020. 5, 6
69. Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan. [Principle-driven self-alignment of language models from scratch with minimal human supervision](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/). arXiv preprint arXiv:2305.03047, 2023. 4
70. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. [Intriguing properties of neural networks](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/). arXiv preprint arXiv:1312.6199, 2013. 1, 4
71. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. 6, 16
72. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023. 3, 6, 16
73. F. Tramer. Detecting adversarial examples is (nearly) as hard as classifying them. In International Conference on Machine Learning, pages 21692–21702. PMLR, 2022. 1, 4, 9 [find](https://scholar.google.com/scholar?q=F%2E+Tramer%2E+Detecting+adversarial+examples+is+%28nearly%29+as+hard+as+classifying+them%2E+In+International+Conference+on+Machine+Learning%2C+pages+21692%E2%80%9321702%2E+PMLR%2C+2022%2E+1%2C+4%2C+9)
74. E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125, 2019. 6, 10 [find](https://scholar.google.com/scholar?q=E%2E+Wallace%2C+S%2E+Feng%2C+N%2E+Kandpal%2C+M%2E+Gardner%2C+and+S%2E+Singh%2E+Universal+adversarial+triggers+for+attacking+and+analyzing+nlp%2E+arXiv+preprint+arXiv%3A1908%2E07125%2C+2019%2E+6%2C+10)
75. A. Wei, N. Haghtalab, and J. Steinhardt. [Jailbroken: How does llm safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) arXiv:2307.02483, 2023. 4
76. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652, 2021. 3
77. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022. 3
78. L. Weng, V. Goel, and A. Vallone. Using gpt-4 for content moderation, 8 2023. URL [https://openai.com/blo](https://openai.com/blo) g/using-gpt-4-for-content-moderation . 10 [↗](https://openai.com/blo)
79. C. Xiang, S. Mahloujifar, and P. Mittal. {PatchCleanser}: Certifiably robust defense against adversarial patches for any image classifier. In 31st USENIX Security Symposium (USENIX Security 22), pages 2065–2082, 2022. 10 [find](https://scholar.google.com/scholar?q=C%2E+Xiang%2C+S%2E+Mahloujifar%2C+and+P%2E+Mittal%2E+%7BPatchCleanser%7D%3A+Certifiably+robust+defense+against+adversarial+patches+for+any+image+classifier%2E+In+31st+USENIX+Security+Symposium+%28USENIX+Security+22%29%2C+pages)
80. R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019. 1 [find](https://scholar.google.com/scholar?q=R%2E+Zellers%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+recognition%2C+pages+6)
81. K. Zenner. A law for foundation models: the eu ai act can improve regulation for fairer competition. [https://oecd.ai/en/wonk/foundation-models-eu-ai-act-fairer-competition](https://oecd.ai/en/wonk/foundation-models-eu-ai-act-fairer-competition) , July 2023. the Head of Office and Digital Policy Adviser, Office of Axel Voss MEP. 11 [↗](https://oecd.ai/en/wonk/foundation-models-eu-ai-act-fairer-competition)
82. Z. Zhao, D. Dua, and S. Singh. Generating natural adversarial examples. arXiv preprint arXiv:1710.11342, 2017. 1, 4 [find](https://scholar.google.com/scholar?q=Z%2E+Zhao%2C+D%2E+Dua%2C+and+S%2E+Singh%2E+Generating+natural+adversarial+examples%2E+arXiv+preprint+arXiv%3A1710%2E11342%2C+2017%2E+1%2C+4)
83. D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023. 2, 3, 6, 16
84. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043, 2023. 4 arXiv preprint

*Updated 2 November 2026*