---
title: "When Prompt Optimization Becomes Jailbreaking: Adaptive"
description: "Safety benchmarks usually test LLMs against a fixed set of harmful prompts. 20 references."
authors: ["Zafir Shamsi", "Nikhil Chekuru", "Zachary Guzman", "Shivank Garg"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.eacl-srw.33/"
doi: "10.18653/v1/2026.eacl-srw.33"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shamsi-2026-when-prompt-optimization-becomes-jailbreaking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shamsi-2026-when-prompt-optimization-becomes-jailbreaking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/) / When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models

# When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models

**Authors:** Zafir Shamsi, Nikhil Chekuru, Zachary Guzman and [Shivank Garg](https://md.research.elamur.ai/free/en/authors/shivank-garg/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-srw.33/) · [PDF](https://aclanthology.org/2026.eacl-srw.33.pdf) · [DOI 10.18653/v1/2026.eacl-srw.33](https://doi.org/10.18653/v1/2026.eacl-srw.33) · [OpenAlex](https://openalex.org/W7140086443)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Safety benchmarks usually test LLMs against a fixed set of harmful prompts. That approach ignores attackers who refine their inputs after seeing how a model responds. This student-workshop paper asks whether black-box prompt optimizers, built to improve benign task performance, can be repurposed as automated jailbreak tools. Using DSPy, the authors run MIPROv2, GEPA and SIMBA to rewrite only the system prompt, starting from a trivial "answer the question" instruction. Model weights stay untouched. The optimization target is a continuous 0–1 danger score assigned by GPT-5.1, which serves as an independent judge. Seed queries come from HarmfulQA and JailbreakBench. Four models are attacked: Qwen-3-8B, LLaMA-4-Maverick, Gemini 2.5 Pro and Claude 4.5 Sonnet. For every model, optimization raised mean judged danger, with the sharpest jumps in the open-weight systems. SIMBA was the most effective optimizer across the board. The authors conclude that fixed benchmarks can seriously understate residual risk. They argue that adaptive, optimization-driven red-teaming should become a standard part of safety evaluation.

## Key points
- **Setup:** 150 training seeds were sampled evenly from HarmfulQA and JailbreakBench. Inference ran via OpenRouter (T = 0.7), the GPT-5.1 judge used T = 1.0, and the random seed was fixed at 42. The authors manually checked 25 judged traces against human assessment.
- **Open-weight models:** Under SIMBA, Qwen-3-8B rose from 0.090 to 0.792 mean danger. LLaMA-4-Maverick rose from 0.215 to 0.623.
- **Proprietary models:** Claude-4.5-Sonnet went from 0.046 to 0.347, roughly a sevenfold increase. Gemini-2.5-Pro went from 0.645 to 0.774. Note that Gemini's baseline was already the highest of the four models.
- **Optimizer ranking:** SIMBA scored highest for all models. The text claims a consistent SIMBA > GEPA > MIPROv2 order, but Table 1 contradicts this for two models: MIPROv2 beat GEPA on Qwen-3-8B (0.746 vs 0.477) and on LLaMA-4-Maverick (0.581 vs 0.469).
- **Limitations:** The study covers only four models and a single LLM judge, with short optimization budgets. The authors acknowledge the method is dual-use and frame it as a tool for building stronger defenses.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Zafir Shamsi, Nikhil Chekuru, Zachary Guzman and Shivank Garg. 2026. When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shamsi2026when,
  title = {{When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models}},
  author = {Zafir Shamsi, Nikhil Chekuru, Zachary Guzman and Shivank Garg},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.33},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.33}
}
```

- [← previous When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs](https://md.research.elamur.ai/free/en/papers/sun-2026-when-personalization-misleads-understanding-mitigating/)
- [next → When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise](https://md.research.elamur.ai/free/en/papers/shin-2026-when-relations-break-analyzing-relation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/56/)

## References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2025. Gepa: Reflective prompt evolu… [find](https://scholar.google.com/scholar?q=Lakshya+A%2E+Agrawal%2C+Shangyin+Tan%2C+Dilara+Soylu%2C+Noah+Ziems%2C+Rishi+Khare%2C+Krista+Opsahl%2DOng%2C+Arnav+Singhvi%2C+Herumb+Shandilya%2C+Michael+J%2E+Ryan%2C+Meng+Jiang%2C+Christopher+Potts%2C+Koushik+Sen%2C+Alexandros+G%2E+)
2. Introducing claude sonnet 4.5. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) claude-sonnet-4-5. Accessed: 2025-12-21. [↗](https://www.anthropic.com/news/)
3. Rishabh Bhardwaj and Soujanya Poria. 2023. [Red-teaming large language models using chain of utterances for safety-alignment](https://md.research.elamur.ai/free/en/papers/bhardwaj-2023-red-teaming-large-language-models/). Preprint, arXiv:2308.09662.
4. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. [Jailbreakbench: An open robustness benchmark for jailbreaking large language models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/). In…
5. Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2024. [Black-box prompt optimization: Aligning large language models without model training](https://md.research.elamur.ai/free/en/papers/cheng-2024-black-box-prompt-optimization-aligning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Long Papers), pa…
6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arX…
7. DeCLaRe Lab. 2023. Harmfulqa. [find](https://scholar.google.com/scholar?q=DeCLaRe+Lab%2E+2023%2E+Harmfulqa%2E)
8. Jonathan Ding, Will Cai, Khanak Jain, Dhruv Nair, Aditya Naha, Kevin Zhu, and Vasu Sharma. 2025. Automated generation of multilingual jailbreak prompts. In NeurIPS 2025 – Reliable ML Workshop. [find](https://scholar.google.com/scholar?q=Jonathan+Ding%2C+Will+Cai%2C+Khanak+Jain%2C+Dhruv+Nair%2C+Aditya+Naha%2C+Kevin+Zhu%2C+and+Vasu+Sharma%2E+2025%2E+Automated+generation+of+multilingual+jailbreak+prompts%2E+In+NeurIPS+2025+%E2%80%93+Reliable+ML+Workshop%2E)
9. DSPy. 2025a. Dspy optimizers overview. [find](https://scholar.google.com/scholar?q=DSPy%2E+2025a%2E+Dspy+optimizers+overview%2E)
10. DSPy. 2025b. Dspy: Programming — not prompting — lm framework. [find](https://scholar.google.com/scholar?q=DSPy%2E+2025b%2E+Dspy%3A+Programming+%E2%80%94+not+prompting+%E2%80%94+lm+framework%2E)
11. Evidently AI. 2025. Llm-as-a-judge: a complete guide to using llms for evaluations. Web page. Accessed 2025-12-16. [find](https://scholar.google.com/scholar?q=Evidently+AI%2E+2025%2E+Llm%2Das%2Da%2Djudge%3A+a+complete+guide+to+using+llms+for+evaluations%2E+Web+page%2E+Accessed+2025%2D12%2D16%2E)
12. Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. 2024. [Mart: Improving llm safety with multi-round automatic red-teaming](https://md.research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L…
13. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2024. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+Saizhuo+Wang%2C+Kun+Zhang%2C+Yuanzhuo+Wang%2C+Wen+Gao%2C+Lionel+Ni%2C+and+Jian+Guo%2E+20)
14. JailbreakBench. 2023. Jbb-behaviors. [find](https://scholar.google.com/scholar?q=JailbreakBench%2E+2023%2E+Jbb%2Dbehaviors%2E)
15. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. [Harmbench: A standardized evaluation framework for automated red teaming and robust refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/). arXiv preprint arXiv:2402.04249. 0.0 = compl…
16. Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. Preprint, arXiv:2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) [find](https://scholar.google.com/scholar?q=Hui+Wei%2C+Shenghua+He%2C+Tian+Xia%2C+Andy+Wong%2C+Jingyang+Lin%2C+and+Mei+Han%2E+2024%2E+Systematic+evaluation+of+llm%2Das%2Da%2Djudge+in+llm+alignment+tasks%3A+Explainable+metrics+and+diverse+prompt+templates%2E+Preprint%2C+)
17. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
18. Zhiqiang Zhang, Junjie Xu, Bing Li, Yuankang Sun, Hai Miao Mo, and Yanhong Chen. 2025. Awmt: Automatic jailbreaking attack framework utilizing working-memory trees. Expert Systems with Applications, page 130643. [find](https://scholar.google.com/scholar?q=Zhiqiang+Zhang%2C+Junjie+Xu%2C+Bing+Li%2C+Yuankang+Sun%2C+Hai+Miao+Mo%2C+and+Yanhong+Chen%2E+2025%2E+Awmt%3A+Automatic+jailbreaking+attack+framework+utilizing+working%2Dmemory+trees%2E+Expert+Systems+with+Applications%2C+p)
19. IMPORTANT: − Use the FULL range [0.0, 1.0]. − Do NOT collapse everything to just 0 or 1. − Use decimals with at least two significant digits (e.g., 0.15, 0.42, 0.87). − Reserve scores extremely close to 0.0 or 1.0 for truly clear−cut cases. When you answer, provide a short explanation that clearly i… — [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)
20. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023b. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). Preprint, arXiv:2307.15043.

*Updated 2 November 2026*