---
title: "ZIP: Quantifying Which Words Matter in Zero-Shot — Sadr 2026"
description: "Zero-shot instructional prompts such as the Chain-of-Thought cue \"Let's think step-by-step\" reliably improve LLM accuracy, but it has been unclear which…"
authors: ["Nikta Gohari Sadr", "Sangmitra Madhusudan", "Arash Asgari", "Hassan Sajjad", "Laleh Seyyed-Kalantari", "Ali Emami"]
year: 2026
venue: "Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)"
source: "https://aclanthology.org/2026.starsem-conference.30/"
doi: "10.18653/v1/2026.starsem-conference.30"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sadr-2026-zip-quantifying-which-words-matter/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sadr-2026-zip-quantifying-which-words-matter/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / ZIP: Quantifying Which Words Matter in Zero-Shot Instructional Prompts

# ZIP: Quantifying Which Words Matter in Zero-Shot Instructional Prompts

**Authors:** Nikta Gohari Sadr, Sangmitra Madhusudan, [Arash Asgari](https://md.research.elamur.ai/free/en/authors/arash-asgari/), [Hassan Sajjad](https://md.research.elamur.ai/free/en/authors/hassan-sajjad/), [Laleh Seyyed-Kalantari](https://md.research.elamur.ai/free/en/authors/laleh-seyyed-kalantari/), [Ali Emami](https://md.research.elamur.ai/free/en/authors/ali-emami/) · **Institutions:** [Emory University](https://md.research.elamur.ai/free/en/universities/emory-university/) · [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) · [York University](https://md.research.elamur.ai/free/en/universities/york-university/) · [Dalhousie University](https://md.research.elamur.ai/free/en/universities/dalhousie-university/) · [York University](https://md.research.elamur.ai/free/en/universities/york-university-us/) · Brock University · **Year:** 2026 · **Venue:** Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (\*SEM 2026) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.starsem-conference.30/) · [PDF](https://aclanthology.org/2026.starsem-conference.30.pdf) · [DOI 10.18653/v1/2026.starsem-conference.30](https://doi.org/10.18653/v1/2026.starsem-conference.30) · [OpenAlex](https://openalex.org/W7166843729)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Zero-shot instructional prompts such as the Chain-of-Thought cue "Let's think step-by-step" reliably improve LLM accuracy, but it has been unclear which individual words produce the gain. The authors propose the ZIP score (Zero-shot Importance of Perturbation). It is a black-box measure that needs only query access, so it also works on closed models.

For each word in a prompt, GPT-4 proposes synonym and co-hyponym substitutes, and a deletion variant is added. A candidate is kept only if its Universal Sentence Encoder similarity to the original exceeds 30% and GPT-4 judges it grammatical and coherent. A word's importance reflects how much the model's answers shift under these edits, and statistical testing flags the significant words.

To verify attributions, the paper introduces what it describes as the first ground-truth benchmark for prompt interpretability. Its prompts make the correct output depend on a single known keyword, and ZIP recovers that keyword much more reliably than LIME.

Applied to seven prompts, six flagship LLMs and several task domains, ZIP shows that important words differ by task and by model family. Prompts also matter most where models perform poorly. This offers a principled basis for prompt design.

## Key points
- On the ground-truth benchmark, ZIP finds the key word with 95.8% accuracy, compared with 65.8% for LIME.
- Word importance depends on the task. "step-by-step" matters most for mathematical reasoning (e.g., AQUA-RAT), while "think" matters more for common-sense tasks.
- Higher ZIP scores accompany lower model performance, a strongly negative relationship (|r| > 0.9).
- Measured with Jensen-Shannon similarity, GPT-4o and Llama-3 agree most closely with human judgments of word importance, and Llama-2 and Gemini-2.0 agree least.
- Across models, nouns make up 47%–66% of significant words. GPT-4o, GPT-3.5 and Gemini-2.0 have similar part-of-speech profiles, while open-source models give more weight to verbs and adverbs.
- The pipeline averages about 9.04 valid perturbations per word and uses fewer than 2,150 tokens for the CoT prompt.
- Small edits can break reasoning: replacing "solve" with "work out" in Plan-and-Solve caused GPT-4o to fail an arithmetic problem.
- Code is available at github.com/niktaas/ZIP.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sadr2026zip,
  title = {{ZIP: Quantifying Which Words Matter in Zero-Shot Instructional Prompts}},
  author = {Sadr, Nikta Gohari and Madhusudan, Sangmitra and Asgari, Arash and Sajjad, Hassan and Seyyed-Kalantari, Laleh and Emami, Ali},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.30},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.30}
}
```

- [← previous Yoysef Meyer Yavets (1832–1914), Yiddish Translator of Sacred Jewish Books](https://md.research.elamur.ai/free/en/papers/hever-2026-chybowski-yoysef-meyer-yavets-18321914-yiddish/)
- [next → А.В. Раукова (A.V. Paykova), Легенды и сказания в памятниках сирийской агиографии Legends and Tales in the Monuments of Syriac Hagiographic Literature, Nauka, Academy of Sciences of the USSR, Palestinskij Sbrnik, Vol. 30 (93), Leningrad 1990, pp. 142](https://md.research.elamur.ai/free/en/papers/bieniek-2026-v-paykova-legends-tales-monuments/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/)

## References

47 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eshaan Agarwal, Raghav Magazine, Joykirat Singh, Vivek Dani, Tanuja Ganu, and Akshay Nambi. 2025. PromptWizard: Optimizing prompts via task-aware, feedback-driven self-evolution. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19974–20003, Vienna, Austria. Association f… [find](https://scholar.google.com/scholar?q=Eshaan+Agarwal%2C+Raghav+Magazine%2C+Joykirat+Singh%2C+Vivek+Dani%2C+Tanuja+Ganu%2C+and+Akshay+Nambi%2E+2025%2E+PromptWizard%3A+Optimizing+prompts+via+task%2Daware%2C+feedback%2Ddriven+self%2Devolution%2E+In+Findings+of+the+As)
2. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (… [find](https://scholar.google.com/scholar?q=Lo%C3%AFc+Barrault%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Shervin+Malmasi%2C+Christof+Monz%2C+Mathias+M%C3%BCller%2C+Santanu) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
3. Jasmijn Bastings and Katja Filippova. 2020. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149–155, Online. Association for Comput… [find](https://scholar.google.com/scholar?q=Jasmijn+Bastings+and+Katja+Filippova%2E+2020%2E+The+elephant+in+the+interpretability+room%3A+Why+use+attention+as+explanation+when+we+have+saliency+methods%3F+In+Proceedings+of+the+Third+BlackboxNLP+Workshop+)
4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+language+processing+with+Python%3A+analyzing+text+with+the+natural+language+toolkit%2E+%22+O%E2%80%99Reilly+Media%2C+Inc%2E%22%2E)
5. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, and 1 others. 2018. Universal sentence encoder for english. In Proceedings of the 2018 conference on empirical methods in natural language processing: syst… [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Yinfei+Yang%2C+Sheng%2Dyi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+Chris+Tar%2C+and+1+others%2E+2018%2E+Universal+sentence+encoder+for+engl) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder-english/)
6. Shivani Choudhary, Niladri Chatterjee, and Subir Kumar Saha. 2022. Interpretation of black box nlp models: A survey. arXiv preprint arXiv:2203.17081. — [arXiv:2203.17081](https://arxiv.org/abs/2203.17081) [find](https://scholar.google.com/scholar?q=Shivani+Choudhary%2C+Niladri+Chatterjee%2C+and+Subir+Kumar+Saha%2E+2022%2E+Interpretation+of+black+box+nlp+models%3A+A+survey%2E+arXiv+preprint+arXiv%3A2203%2E17081%2E+%E2%80%94+%5BarXiv%3A2203%2E17081%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2203%2E17)
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+2021%2E+Train) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX… [find](https://scholar.google.com/scholar?q=Gheorghe+Comanici%2C+Eric+Bieber%2C+Mike+Schaekermann%2C+Ice+Pasupat%2C+Noveen+Sachdeva%2C+Inderjit+Dhillon%2C+Marcel+Blistein%2C+Ori+Ram%2C+Dan+Zhang%2C+Evan+Rosen%2C+and+1+others%2E+2025%2E+Gemini+2%2E5%3A+Pushing+the+frontier) [our card](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
9. Kawin Ethayarajh and Dan Jurafsky. 2021. Attention flows are shapley value explanations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 49–54, Online… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh+and+Dan+Jurafsky%2E+2021%2E+Attention+flows+are+shapley+value+explanations%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Interna)
10. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. Associatio… [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+neural+models+make+interpretations+difficult%2E+In+Proceedings+of+the+2018+Conference)
11. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2024. Promptbreeder: self-referential self-improvement via prompt evolution. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. [find](https://scholar.google.com/scholar?q=Chrisantha+Fernando%2C+Dylan+Banarse%2C+Henryk+Michalewski%2C+Simon+Osindero%2C+and+Tim+Rockt%C3%A4schel%2E+2024%2E+Promptbreeder%3A+self%2Dreferential+self%2Dimprovement+via+prompt+evolution%2E+In+Proceedings+of+the+41st+Int) [our card](https://md.research.elamur.ai/free/en/papers/fernando-2023-promptbreeder-self-referential-self-improvement/)
12. Javier Ferrando, Gerard I. Gállego, and Marta R. Costa-jussà. 2022. Measuring the mixing of contextual information in the transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8698–8714, Abu Dhabi, United Arab Emirates. Association for Computat… [find](https://scholar.google.com/scholar?q=Javier+Ferrando%2C+Gerard+I%2E+G%C3%A1llego%2C+and+Marta+R%2E+Costa%2Djuss%C3%A0%2E+2022%2E+Measuring+the+mixing+of+contextual+information+in+the+transformer%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Nat)
13. Javier Ferrando, Gerard I. Gállego, Ioannis Tsiamas, and Marta R. Costa-jussà. 2023. Explaining how transformers use context to build predictions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5513, Toronto, Canada. Ass… [find](https://scholar.google.com/scholar?q=Javier+Ferrando%2C+Gerard+I%2E+G%C3%A1llego%2C+Ioannis+Tsiamas%2C+and+Marta+R%2E+Costa%2Djuss%C3%A0%2E+2023%2E+Explaining+how+transformers+use+context+to+build+predictions%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Asso)
14. Stefan Hackmann, Haniyeh Mahmoudian, Mark Stead-man, and Michael Schmidt. 2024. Word importance explains how prompts affect language model outputs. arXiv preprint arXiv:2403.03028. — [arXiv:2403.03028](https://arxiv.org/abs/2403.03028) [find](https://scholar.google.com/scholar?q=Stefan+Hackmann%2C+Haniyeh+Mahmoudian%2C+Mark+Stead%2Dman%2C+and+Michael+Schmidt%2E+2024%2E+Word+importance+explains+how+prompts+affect+language+model+outputs%2E+arXiv+preprint+arXiv%3A2403%2E03028%2E+%E2%80%94+%5BarXiv%3A2403%2E03028)
15. Ari Holtzman and Chenhao Tan. 2025. Prompting as scientific inquiry. arXiv preprint arXiv:2507.00163. — [arXiv:2507.00163](https://arxiv.org/abs/2507.00163) [find](https://scholar.google.com/scholar?q=Ari+Holtzman+and+Chenhao+Tan%2E+2025%2E+Prompting+as+scientific+inquiry%2E+arXiv+preprint+arXiv%3A2507%2E00163%2E+%E2%80%94+%5BarXiv%3A2507%2E00163%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2507%2E00163%29)
16. Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. 2021. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters, 150:228–234. [find](https://scholar.google.com/scholar?q=Maksims+Ivanovs%2C+Roberts+Kadikis%2C+and+Kaspars+Ozols%2E+2021%2E+Perturbation%2Dbased+methods+for+explaining+deep+neural+networks%3A+A+survey%2E+Pattern+Recognition+Letters%2C+150%3A228%E2%80%93234%2E)
17. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa… [find](https://scholar.google.com/scholar?q=Sarthak+Jain+and+Byron+C%2E+Wallace%2E+2019%2E+Attention+is+not+Explanation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Langu) [our card](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/)
18. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
19. Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. 2020. Interpretation of NLP models through input marginalization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3154–3167, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Siwon+Kim%2C+Jihun+Yi%2C+Eunji+Kim%2C+and+Sungroh+Yoon%2E+2020%2E+Interpretation+of+NLP+models+through+input+marginalization%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
20. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2024. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+2024%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+In+Proceedings+of+the+36th+International+Conference+on+Neural+In) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
21. Bei Li, Rui Wang, Junliang Guo, Kaitao Song, Xu Tan, Hany Hassan, Arul Menezes, Tong Xiao, Jiang Bian, and JingBo Zhu. 2023. Deliberate then generate: Enhanced prompting framework for text generation. Preprint, arXiv:2305.19835. — [arXiv:2305.19835](https://arxiv.org/abs/2305.19835) [find](https://scholar.google.com/scholar?q=Bei+Li%2C+Rui+Wang%2C+Junliang+Guo%2C+Kaitao+Song%2C+Xu+Tan%2C+Hany+Hassan%2C+Arul+Menezes%2C+Tong+Xiao%2C+Jiang+Bian%2C+and+JingBo+Zhu%2E+2023%2E+Deliberate+then+generate%3A+Enhanced+prompting+framework+for+text+generation%2E)
22. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 1 others… [find](https://scholar.google.com/scholar?q=Jack+Lindsey%2C+Wes+Gurnee%2C+Emmanuel+Ameisen%2C+Brian+Chen%2C+Adam+Pearce%2C+Nicholas+L%2E+Turner%2C+Craig+Citro%2C+David+Abrahams%2C+Shan+Carter%2C+Basil+Hosmer%2C+Jonathan+Marcus%2C+Michael+Sklar%2C+Adly+Templeton%2C+Trenton)
23. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation : Learning to solve and explain algebraic word problems. Preprint, arXiv:1705.04146. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation+%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+Preprint%2C+arXiv%3A1705%2E04146%2E)
24. Fuxiao Liu, Paiheng Xu, Zongxia Li, Yue Feng, and Hyemi Song. 2023. Towards understanding in-context learning with contrastive demonstrations and saliency maps. arXiv preprint arXiv:2307.05052. — [arXiv:2307.05052](https://arxiv.org/abs/2307.05052) [find](https://scholar.google.com/scholar?q=Fuxiao+Liu%2C+Paiheng+Xu%2C+Zongxia+Li%2C+Yue+Feng%2C+and+Hyemi+Song%2E+2023%2E+Towards+understanding+in%2Dcontext+learning+with+contrastive+demonstrations+and+saliency+maps%2E+arXiv+preprint+arXiv%3A2307%2E05052%2E+%E2%80%94+%5BarX)
25. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2022%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
26. Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. 2022. Globenc: Quantifying global token attribution by incorporating the whole encoder layer in transformers. arXiv preprint arXiv:2205.03286. [find](https://scholar.google.com/scholar?q=Ali+Modarressi%2C+Mohsen+Fayyaz%2C+Yadollah+Yaghoobzadeh%2C+and+Mohammad+Taher+Pilehvar%2E+2022%2E+Globenc%3A+Quantifying+global+token+attribution+by+incorporating+the+whole+encoder+layer+in+transformers%2E+arXiv+p)
27. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+Gpt%2D4+technical+report%2E+arXiv%3A2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
28. Preprint, OpenAI. 2025. OpenAI o3-mini. [https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/). Accessed: 2025-01-01. [↗](https://openai.com/index/openai-o3-mini/)
29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
30. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. Preprint, arXiv:2303.13780. [find](https://scholar.google.com/scholar?q=Keqin+Peng%2C+Liang+Ding%2C+Qihuang+Zhong%2C+Li+Shen%2C+Xuebo+Liu%2C+Min+Zhang%2C+Yuanxin+Ouyang%2C+and+Dacheng+Tao%2E+2023%2E+Towards+making+the+most+of+chatgpt+for+machine+translation%2E+Preprint%2C+arXiv%3A2303%2E13780%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-making-most-chatgpt-machine-translation/)
31. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135– 1144. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016%2E+%22+why+should+i+trust+you%3F%22+explaining+the+predictions+of+any+classifier%2E+In+Proceedings+of+the+22nd+ACM+SIGKDD+international+conference+on) [our card](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/)
32. Rabia Saleem, Bo Yuan, Fatih Kurugollu, Ashiq Anjum, and Lu Liu. 2022. Explaining deep neural networks: A survey on the global interpretation methods. Neurocomputing, 513:165–180. [find](https://scholar.google.com/scholar?q=Rabia+Saleem%2C+Bo+Yuan%2C+Fatih+Kurugollu%2C+Ashiq+Anjum%2C+and+Lu+Liu%2E+2022%2E+Explaining+deep+neural+networks%3A+A+survey+on+the+global+interpretation+methods%2E+Neurocomputing%2C+513%3A165%E2%80%93180%2E)
33. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2023%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+i+learned+to+start+worrying+about+prompt+formatt)
34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+V+Le%2C+Denny+Zhou%2C+and+1+others%2E+2022%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+Advances+in+n) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
35. Chirag Shah. 2025. From prompt engineering to prompt science with humans in the loop. Commun. ACM, 68(6):54–61. [find](https://scholar.google.com/scholar?q=Chirag+Shah%2E+2025%2E+From+prompt+engineering+to+prompt+science+with+humans+in+the+loop%2E+Commun%2E+ACM%2C+68%286%29%3A54%E2%80%9361%2E)
36. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. Preprint, arXiv:2309.03409. [find](https://scholar.google.com/scholar?q=Chengrun+Yang%2C+Xuezhi+Wang%2C+Yifeng+Lu%2C+Hanxiao+Liu%2C+Quoc+V%2E+Le%2C+Denny+Zhou%2C+and+Xinyun+Chen%2E+2024%2E+Large+language+models+as+optimizers%2E+Preprint%2C+arXiv%3A2309%2E03409%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)
37. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. Preprint, arXiv:2302.00093. [find](https://scholar.google.com/scholar?q=Freda+Shi%2C+Xinyun+Chen%2C+Kanishka+Misra%2C+Nathan+Scales%2C+David+Dohan%2C+Ed+Chi%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Large+language+models+can+be+easily+distracted+by+irrelevant+context%2E+Preprint%2C+arX) [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/)
38. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, and 1 others. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Preprint, arXiv:2206.04615. [find](https://scholar.google.com/scholar?q=Aarohi+Srivastava%2C+Abhinav+Rastogi%2C+Abhishek+Rao%2C+Abu+Awal+Md+Shoeb%2C+Abubakar+Abid%2C+Adam+Fisch%2C+and+1+others%2E+2023%2E+Beyond+the+imitation+game%3A+Quantifying+and+extrapolating+the+capabilities+of+languag) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
39. Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and 1 others. 2020. The language interpretability tool: Extensible, interactive visualizations and analysis for nlp models. arXiv preprint arXiv:2… [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+James+Wexler%2C+Jasmijn+Bastings%2C+Tolga+Bolukbasi%2C+Andy+Coenen%2C+Sebastian+Gehrmann%2C+Ellen+Jiang%2C+Mahima+Pushkarna%2C+Carey+Radebaugh%2C+Emily+Reif%2C+and+1+others%2E+2020%2E+The+language+interpretabil)
40. Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Jason Wu. 2023. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. arXiv preprint arXiv:2306.01150. [find](https://scholar.google.com/scholar?q=Fan+Yin%2C+Jesse+Vig%2C+Philippe+Laban%2C+Shafiq+Joty%2C+Caiming+Xiong%2C+and+Chien%2DSheng+Jason+Wu%2E+2023%2E+Did+you+read+the+instructions%3F+rethinking+the+effectiveness+of+task+definitions+in+instruction+learning%2E)
41. Kayo Yin and Graham Neubig. 2022. Interpreting language models with contrastive explanations. arXiv preprint arXiv:2202.10419. [find](https://scholar.google.com/scholar?q=Kayo+Yin+and+Graham+Neubig%2E+2022%2E+Interpreting+language+models+with+contrastive+explanations%2E+arXiv+preprint+arXiv%3A2202%2E10419%2E)
42. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. Preprint, arXiv:2211.01910. [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Andrei+Ioan+Muresanu%2C+Ziwen+Han%2C+Keiran+Paster%2C+Silviu+Pitis%2C+Harris+Chan%2C+and+Jimmy+Ba%2E+2023%2E+Large+language+models+are+human%2Dlevel+prompt+engineers%2E+Preprint%2C+arXiv%3A2211%2E01910%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/)
43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+and+1+others%2E+2023%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+Prepri)
44. Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019. AllenNLP interpret: A framework for explaining predictions of NLP models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confere… [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Jens+Tuyls%2C+Junlin+Wang%2C+Sanjay+Subramanian%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+AllenNLP+interpret%3A+A+framework+for+explaining+predictions+of+NLP+models%2E+In+Proceedings+of+the+2019+Co)
45. Junlin Wang, Jens Tuyls, Eric Wallace, and Sameer Singh. 2020. Gradient-based analysis of nlp models is manipulable. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 247–258. [find](https://scholar.google.com/scholar?q=Junlin+Wang%2C+Jens+Tuyls%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+Gradient%2Dbased+analysis+of+nlp+models+is+manipulable%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+247)
46. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. Preprint, arXiv:2305.04091. [find](https://scholar.google.com/scholar?q=Lei+Wang%2C+Wanyu+Xu%2C+Yihuai+Lan%2C+Zhiqiang+Hu%2C+Yunshi+Lan%2C+Roy+Ka%2DWei+Lee%2C+and+Ee%2DPeng+Lim%2E+2023a%2E+Plan%2Dand%2Dsolve+prompting%3A+Improving+zero%2Dshot+chain%2Dof%2Dthought+reasoning+by+large+language+models%2E+Prep)
47. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171. [find](https://scholar.google.com/scholar?q=Xuezhi+Wang%2C+Jason+Wei%2C+Dale+Schuurmans%2C+Quoc+Le%2C+Ed+Chi%2C+Sharan+Narang%2C+Aakanksha+Chowdhery%2C+and+Denny+Zhou%2E+2023b%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+Preprint%2C+) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)