---
title: "Beyond Prompt Engineering — paper by Xie et al. (2026)"
description: "The authors build both features into an automated Prompt-Refining Agent, which cuts performance variance on code… 25 references."
authors: ["Qipeng Xie", "Zi Liang", "Jiafei Wu", "Yufei Chen", "Weizheng Wang", "Wenao Ma", "Zhong Ming", "Haiqin Yang", "Kaishun Wu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.2084/"
doi: "10.18653/v1/2026.findings-acl.2084"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xie-2026-beyond-prompt-engineering-systematic-analysis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xie-2026-beyond-prompt-engineering-systematic-analysis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

# Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

**Authors:** Qipeng Xie, [Zi Liang](https://md.research.elamur.ai/free/en/authors/zi-liang/), Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, [Haiqin Yang](https://md.research.elamur.ai/free/en/authors/haiqin-yang/) and [Kaishun Wu](https://md.research.elamur.ai/free/en/authors/kaishun-wu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.2084/) · [PDF](https://aclanthology.org/2026.findings-acl.2084.pdf) · [DOI 10.18653/v1/2026.findings-acl.2084](https://doi.org/10.18653/v1/2026.findings-acl.2084) · [OpenAlex](https://openalex.org/W7166800784)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Large language models often react strongly to small wording changes in a prompt. Prior work mostly treats this as a black-box optimization problem or examines whole templates, so it does not explain which words make a prompt dependable. The authors instead analyze prompt sensitivity statistically at the n-gram token level. They draw 12,000 English instructions from WizardLM's benchmark, spanning programming, mathematics and creative writing tasks, and generate 11 variants per instruction, yielding 132,000 evaluated prompts. The variants come from Gemini-2.5-flash using five meaning-preserving but stylistically different rewriting strategies. The analysis reveals a scaling-law-like pattern: prompts with higher average scores also fluctuate less across rewrites. Two lexical features account for much of this stability. Domain-specific terminology fixes the task's semantic scope, and explicit action directives spell out what the model must do. The authors build both features into an automated Prompt-Refining Agent, which cuts performance variance on code generation by 40.7%. The result turns prompt-writing folklore into statistically supported guidance.

## Key points
- **Dataset:** 12,000 instructions × 11 variants = 132,000 prompt formulations. Experiments mainly use the Qwen and Gemini model families, and outputs are scored automatically with a multi-dimensional rubric.
- **Five rewriting strategies:** SEP (paraphrase), ACI (irrelevant added context), CFS (register/style shift), IA (hedging and ambiguity) and SN (typos and spacing noise).
- **Strategy validation:** The strategies were checked with embedding deltas.
  - They form separate PCA clusters, with a mean off-diagonal cosine similarity of about 0.22 (about 0.12 for SN).
  - Their intensity spans roughly 0.07 (IA) to 0.36 (SN) in semantic distance.
- **Practical guidance:** Name the domain explicitly, use concrete operational verbs and formats, and combine both.
- **Limitations:** The study covers only English and the WizardLM tasks, and its findings may differ for smaller or Mixture-of-Experts models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese University of Hong Kong](https://md.research.elamur.ai/free/en/universities/chinese-university-of-hong-kong/) — University, Hong Kong, China · 159 works · 16,154 citations
- [Hong Kong Polytechnic University](https://md.research.elamur.ai/free/en/universities/hong-kong-polytechnic-university/) — University, Hong Kong, China · 106 works · 5,190 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 67 works · 1,765 citations
- [Zhejiang Lab](https://md.research.elamur.ai/free/en/labs/zhejiang-lab/) — Research institute, Hangzhou, China · 9 works · 1,776 citations
- [Shenzhen Technology University](https://md.research.elamur.ai/free/en/universities/shenzhen-technology-university/) — University, Shenzhen, China · 4 works · 59 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

## Cite

Cite as: Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang and Kaishun Wu. 2026. Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xie2026beyond,
  title = {{Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality}},
  author = {Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang and Kaishun Wu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2084},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2084}
}
```

- [← previous Beyond Polarity: Continuous Affect-Enhanced Multimodal Aspect-Based Sentiment Classification](https://md.research.elamur.ai/free/en/papers/meng-2026-beyond-polarity-continuous-affect-enhanced/)
- [next → Beyond Prompt-Sensitive Emotion Words: Stable Embeddings for Tang Poetry Analysis](https://md.research.elamur.ai/free/en/papers/zhang-2026-beyond-prompt-sensitive-emotion-words/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sheng Lu, Hendrik Schuff, and Iryna Gurevych. 2024. How are prompts different in terms of sensitivity? In Proceedings of the North American Chapter of the ACL: Human Language Technologies (NAACL-HLT) 2024, volume 1, pages 5833–5856. [find](https://scholar.google.com/scholar?q=Sheng+Lu%2C+Hendrik+Schuff%2C+and+Iryna+Gurevych%2E+2024%2E+How+are+prompts+different+in+terms+of+sensitivity%3F+In+Proceedings+of+the+North+American+Chapter+of+the+ACL%3A+Human+Language+Technologies+%28NAACL%2DHLT%29+)
2. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). 2022. Fantastical…
3. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). Preprint, arXiv:2308.09583.
4. Andrzej Maćkiewicz and Waldemar Ratajczak. 1993. Principal components analysis (pca). Computers & Geosciences, 19(3):303–342. [find](https://scholar.google.com/scholar?q=Andrzej+Mac%CC%81kiewicz+and+Waldemar+Ratajczak%2E+1993%2E+Principal+components+analysis+%28pca%29%2E+Computers+%26+Geosciences%2C+19%283%29%3A303%E2%80%93342%2E)
5. Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, and Shenghua Liu. 2025. A survey of context engineering for large language models. Preprint, arXiv:2507.13334. — [arXiv:2507.13334](https://arxiv.org/abs/2507.13334) [find](https://scholar.google.com/scholar?q=Lingrui+Mei%2C+Jiayu+Yao%2C+Yuyao+Ge%2C+Yiwei+Wang%2C+Baolong+Bi%2C+Yujun+Cai%2C+Jiazhi+Liu%2C+Mingyu+Li%2C+Zhong%2DZhi+Li%2C+Duzhen+Zhang%2C+Chenlin+Zhou%2C+Jiayi+Mao%2C+Tianze+Xia%2C+Jiafeng+Guo%2C+and+Shenghua+Liu%2E+2025%2E+A+surv)
6. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Preprint, arXiv:2401.00595. [find](https://scholar.google.com/scholar?q=Moran+Mizrahi%2C+Guy+Kaplan%2C+Dan+Malkin%2C+Rotem+Dror%2C+Dafna+Shahaf%2C+and+Gabriel+Stanovsky%2E+2024%2E+State+of+what+art%3F+a+call+for+multi%2Dprompt+llm+evaluation%2E+Preprint%2C+arXiv%3A2401%2E00595%2E)
7. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. [find](https://scholar.google.com/scholar?q=Burt+L+Monroe%2C+Michael+P+Colaresi%2C+and+Kevin+M+Quinn%2E+2008%2E+Fightin%E2%80%99words%3A+Lexical+feature+selection+and+evaluation+for+identifying+the+content+of+political+conflict%2E+Political+Analysis%2C+16%284%29%3A372%E2%80%93403)
8. Hayden Moore and Asfahan Shah. 2025. Evaluating autoformalization robustness via semantically similar paraphrasing. Preprint, arXiv:2511.12784. [find](https://scholar.google.com/scholar?q=Hayden+Moore+and+Asfahan+Shah%2E+2025%2E+Evaluating+autoformalization+robustness+via+semantically+similar+paraphrasing%2E+Preprint%2C+arXiv%3A2511%2E12784%2E)
9. Lilian Ngweta, Kiran Kate, Jason Tsay, and Yara Rizk. 2025. Towards llms robustness to changes in prompt format styles. Preprint, arXiv:2504.06969. [find](https://scholar.google.com/scholar?q=Lilian+Ngweta%2C+Kiran+Kate%2C+Jason+Tsay%2C+and+Yara+Rizk%2E+2025%2E+Towards+llms+robustness+to+changes+in+prompt+format+styles%2E+Preprint%2C+arXiv%3A2504%2E06969%2E)
10. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. [True few-shot learning with language models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/). Preprint, arXiv:2105.11447.
11. Pouya Pezeshkpour and Estevam Hruschka. 2024. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017.
12. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design. In The Twelfth Int. Conf. on Learning Representations, The Twelfth Int. Conf. on Learning Representations. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2024%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design%2E+In+The+Twelfth+Int%2E+Conf%2E+on+Learning+Representations%2C+)
13. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. [Large language models can be easily distracted by irrelevant context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/). Preprint, arXiv:2302.00093.
14. Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind your format: Towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024. [find](https://scholar.google.com/scholar?q=Anton+Voronov%2C+Lena+Wolf%2C+and+Max+Ryabinin%2E+2024%2E+Mind+your+format%3A+Towards+consistent+evaluation+of+in%2Dcontext+learning+improvements%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL)
15. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. Preprint, arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023%2E+Large+language+models+are+not+fair+evaluators%2E+Preprint%2C+arXiv%3A2305%2E17926%2E)
16. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
17. Qiang Yao, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, and Aram Galstyan. 2024. Prompt perturbation consistency learning for robust language models. In Findings of the Association for Computational Linguistics: EACL 2024. [find](https://scholar.google.com/scholar?q=Qiang+Yao%2C+Subhrangshu+Nandi%2C+Ninareh+Mehrabi%2C+Greg+Ver+Steeg%2C+Anoop+Kumar%2C+Anna+Rumshisky%2C+and+Aram+Galstyan%2E+2024%2E+Prompt+perturbation+consistency+learning+for+robust+language+models%2E+In+Findings+of)
18. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. [Flask: Fine-grained language model evaluation based on alignment skill sets](https://md.research.elamur.ai/free/en/papers/ye-2023-flask-fine-grained-language-model/). arXiv preprint arXiv:2307.10928.
19. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. [Making retrieval-augmented language models robust to irrelevant context](https://md.research.elamur.ai/free/en/papers/yoran-2023-making-retrieval-augmented-language-models/). Preprint, arXiv:2310.01558.
20. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International conference on machine learning, pages 12697–12706. PMLR.
21. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yanping Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume 36.
22. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2024. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Han+Zhou%2C+Xingchen+Wan%2C+Lev+Proleev%2C+Diana+Mincu%2C+Jilin+Chen%2C+Katherine+Heller%2C+and+Subhrajit+Roy%2E+2024%2E+Batch+calibration%3A+Rethinking+calibration+for+in%2Dcontext+learning+and+prompt+engineering%2E+In+In)
23. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2024a. [Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts](https://md.research.elamur.ai/free/en/papers/zhu-2023-promptrobust-towards-evaluating-robustness-large/). Preprint, arXiv:2306.04528.
24. Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2024b. Promptbench: A unified library for evaluation of large language models. Preprint, arXiv:2312.07910. — [arXiv:2312.07910](https://arxiv.org/abs/2312.07910) [find](https://scholar.google.com/scholar?q=Kaijie+Zhu%2C+Qinlin+Zhao%2C+Hao+Chen%2C+Jindong+Wang%2C+and+Xing+Xie%2E+2024b%2E+Promptbench%3A+A+unified+library+for+evaluation+of+large+language+models%2E+Preprint%2C+arXiv%3A2312%2E07910%2E+%E2%80%94+%5BarXiv%3A2312%2E07910%5D%28https%3A%2F%2Fa)
25. Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. Prosa: Assessing and understanding the prompt sensitivity of llms. In Findings of the Association for Computational Linguistics: EMNLP 2024. [find](https://scholar.google.com/scholar?q=Jingming+Zhuo%2C+Songyang+Zhang%2C+Xinyu+Fang%2C+Haodong+Duan%2C+Dahua+Lin%2C+and+Kai+Chen%2E+2024%2E+Prosa%3A+Assessing+and+understanding+the+prompt+sensitivity+of+llms%2E+In+Findings+of+the+Association+for+Computatio)

*Updated 2 November 2026*