---
title: "To Write or to Automate Linguistic — Sánchez-Torrón 2026"
description: "LLM output is highly sensitive to prompt design, and in localization workflows prompts are usually refined by hand by language specialists."
authors: ["Marina Sánchez-Torrón", "Daria Akselrod", "Jason Rauchwerk"]
year: 2026
venue: "Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)"
source: "https://aclanthology.org/2026.eamt-1.8/"
licence: "CC BY-ND 4.0 (EAMT)"
licence_url: "https://creativecommons.org/licenses/by-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/torron-2026-write-or-automate-linguistic-prompts/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/torron-2026-write-or-automate-linguistic-prompts/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / To Write or to Automate Linguistic Prompts, That Is the Question

# To Write or to Automate Linguistic Prompts, That Is the Question

**Authors:** Marina Sánchez-Torrón, Daria Akselrod, Jason Rauchwerk · **Year:** 2026 · **Venue:** Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1) · **Licence:** [CC BY-ND 4.0 (EAMT)](https://creativecommons.org/licenses/by-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eamt-1.8/) · [PDF](https://aclanthology.org/2026.eamt-1.8.pdf)

## Summary
LLM output is highly sensitive to prompt design, and in localization workflows prompts are usually refined by hand by language specialists. The authors, from Smartling, test whether automatic prompt optimization can replace this expert work. They compare three setups: zero-shot prompts written by experts, plain DSPy signatures, and DSPy signatures tuned with the GEPA optimizer. These are tested on three production tasks: glossary-driven terminology insertion (post-editing), machine translation, and MQM-style language quality assessment (LQA). The experiments use proprietary data annotated by professional linguists and five model configurations from four providers, each paired with a stronger reflection model.

The results depend on the task. For terminology insertion and translation, GEPA-tuned prompts perform about as well as expert prompts, and most differences are not statistically significant. In LQA, expert prompts detect errors better, while optimized prompts classify error category and severity better. Across tasks, GEPA consistently brings bare DSPy signatures close to expert level. The findings help language service providers decide where to invest effort, although optimization requires labeled data that expert prompting does not.

## Key points
- Model pairs (executor / reflection): GPT-4.1-mini / GPT-4.1, GPT-5.4-mini / GPT-5.4, Gemini 3.1 Flash-Lite / Gemini 3.1 Pro, Claude Sonnet 4.6 / Claude Opus 4.6, and Qwen3:8B-q4 K M / Qwen3:14B-q4 K M. The Qwen models run locally through Ollama.
- Data is split 20% train, 40% validation, and 40% test. Test sets contain 1085 examples (terminology insertion), 989 (translation), and 1250 (LQA). Terminology insertion and LQA cover English into six target locales.
- GEPA runs in its "light" mode with textual gold-vs-predicted feedback. Terminology insertion is optimized separately for BLEU, HTER, and an LLM-as-a-judge score. Significance is tested with a paired stratified bootstrap and Holm–Bonferroni correction.
- Single-stage (unified) systems generally match or beat multi-stage pipelines, by 0.05–0.20 Detection F1 in LQA. The weakest model, Qwen3:8B, shows the largest gain from optimization.
- Limitations: all language pairs are English-centric, evaluation is fully automatic, and only one optimizer is tested. The signatures and results are released on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sancheztorron2026write,
  title = {{To Write or to Automate Linguistic Prompts, That Is the Question}},
  author = {Sánchez-Torrón, Marina and Akselrod, Daria and Rauchwerk, Jason},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.8/}
}
```

- [← previous To Skip, to Swap or to Not Swap? Identifying Step Transition Types in Instructional Manuals](https://md.research.elamur.ai/free/en/papers/yang-2026-skip-swap-or-not-swap/)
- [next → ToneSwiper: Facilitating Manual ToDI-annotation of Dutch Prosody](https://md.research.elamur.ai/free/en/papers/westera-2026-toneswiper-facilitating-manual-todi-annotation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/)

## References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agrawal, Lakshya A, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2025. Gepa: Reflective prompt evolut… [find](https://scholar.google.com/scholar?q=Agrawal%2C+Lakshya+A%2C+Shangyin+Tan%2C+Dilara+Soylu%2C+Noah+Ziems%2C+Rishi+Khare%2C+Krista+Opsahl%2DOng%2C+Arnav+Singhvi%2C+Herumb+Shandilya%2C+Michael+J+Ryan%2C+Meng+Jiang%2C+Christopher+Potts%2C+Koushik+Sen%2C+Alexandros+G%2E+D)
2. Deng, Mingkai, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Goldberg, Yoav, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empi… [find](https://scholar.google.com/scholar?q=Deng%2C+Mingkai%2C+Jianyu+Wang%2C+Cheng%2DPing+Hsieh%2C+Yihan+Wang%2C+Han+Guo%2C+Tianmin+Shu%2C+Meng+Song%2C+Eric+Xing%2C+and+Zhiting+Hu%2E+2022%2E+RLPrompt%3A+Optimizing+discrete+text+prompts+with+reinforcement+learning%2E+In+G) [our card](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
3. Errica, Federico, Davide Sanvito, Giuseppe Siracusano, and Roberto Bifulco. 2025. What did I do wrong? quantifying LLMs’ sensitivity and consistency to prompt engineering. In Chiruzzo, Luis, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter… [find](https://scholar.google.com/scholar?q=Errica%2C+Federico%2C+Davide+Sanvito%2C+Giuseppe+Siracusano%2C+and+Roberto+Bifulco%2E+2025%2E+What+did+I+do+wrong%3F+quantifying+LLMs%E2%80%99+sensitivity+and+consistency+to+prompt+engineering%2E+In+Chiruzzo%2C+Luis%2C+Alan+Ritt)
4. Fernandes, Patrick, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Koehn, Philipp, Barr… [find](https://scholar.google.com/scholar?q=Fernandes%2C+Patrick%2C+Daniel+Deutsch%2C+Mara+Finkelstein%2C+Parker+Riley%2C+Andre%CC%81+Martins%2C+Graham+Neubig%2C+Ankush+Garg%2C+Jonathan+Clark%2C+Markus+Freitag%2C+and+Orhan+Firat%2E+2023%2E+The+devil+is+in+the+errors%3A+Lever) [our card](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/)
5. Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. [find](https://scholar.google.com/scholar?q=Freitag%2C+Markus%2C+George+Foster%2C+David+Grangier%2C+Viresh+Ratnakar%2C+Qijun+Tan%2C+and+Wolfgang+Macherey%2E+2021%2E+Experts%2C+errors%2C+and+context%3A+A+large%2Dscale+study+of+human+evaluation+for+machine+translation%2E+) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/)
6. Guerreiro, Nuno M., Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. https://arxiv.org/abs/2310.10482. [↗](https://arxiv.org/abs/2310.10482) [our card](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
7. Guo, Qingyan, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2025. Evoprompt: Connecting llms with evolutionary algorithms yields powerful prompt optimizers. https://arxiv.org/abs/2309. 08532. [↗](https://arxiv.org/abs/2309)
8. He, Zeyu, Saniya Naphade, and Ting-Hao Kenneth Huang. 2025. Prompting in the dark: Assessing human performance in prompt engineering for data labeling when gold labels are absent. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, page 1–33. ACM, April. [find](https://scholar.google.com/scholar?q=He%2C+Zeyu%2C+Saniya+Naphade%2C+and+Ting%2DHao+Kenneth+Huang%2E+2025%2E+Prompting+in+the+dark%3A+Assessing+human+performance+in+prompt+engineering+for+data+labeling+when+gold+labels+are+absent%2E+In+Proceedings+of+th)
9. Hendy, Amr, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. https://arxiv.org/abs/2302. 09210. [↗](https://arxiv.org/abs/2302)
10. Huang, Xu, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. Lost in the source language: How large language models evaluate the quality of machine translation. In Ku, Lun-Wei, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Huang%2C+Xu%2C+Zhirui+Zhang%2C+Xiang+Geng%2C+Yichao+Du%2C+Jiajun+Chen%2C+and+Shujian+Huang%2E+2024%2E+Lost+in+the+source+language%3A+How+large+language+models+evaluate+the+quality+of+machine+translation%2E+In+Ku%2C+Lun%2DWei)
11. Jiao, Wenxiang, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. https://arxiv.org/abs/2301. 08745. [↗](https://arxiv.org/abs/2301) [our card](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/)
12. Khattab, Omar, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines.… [find](https://scholar.google.com/scholar?q=Khattab%2C+Omar%2C+Arnav+Singhvi%2C+Paridhi+Maheshwari%2C+Zhiyuan+Zhang%2C+Keshav+Santhanam%2C+Sri+Vardhamanan%2C+Saiful+Haq%2C+Ashutosh+Sharma%2C+Thomas+T%2E+Joshi%2C+Hanna+Moazam%2C+Heather+Miller%2C+Matei+Zaharia%2C+and+Chris) [our card](https://md.research.elamur.ai/free/en/papers/khattab-2023-dspy-compiling-declarative-language-model/)
13. Kim, Sejoon, Mingi Sung, Jeonghwan Lee, Hyunkuk Lim, and Jorge Gimenez Perez. 2024. Efficient terminology integration for LLM-based translation in specialized domains. In Haddow, Barry, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation,… [find](https://scholar.google.com/scholar?q=Kim%2C+Sejoon%2C+Mingi+Sung%2C+Jeonghwan+Lee%2C+Hyunkuk+Lim%2C+and+Jorge+Gimenez+Perez%2E+2024%2E+Efficient+terminology+integration+for+LLM%2Dbased+translation+in+specialized+domains%2E+In+Haddow%2C+Barry%2C+Tom+Kocmi%2C+Phi)
14. Kocmi, Tom and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Koehn, Philipp, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore, December. Association for Computa… [find](https://scholar.google.com/scholar?q=Kocmi%2C+Tom+and+Christian+Federmann%2E+2023a%2E+GEMBA%2DMQM%3A+Detecting+translation+quality+error+spans+with+GPT%2D4%2E+In+Koehn%2C+Philipp%2C+Barry+Haddow%2C+Tom+Kocmi%2C+and+Christof+Monz%2C+editors%2C+Proceedings+of+the+E) [our card](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/)
15. Kocmi, Tom and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. https://arxiv.org/abs/ 2302.14520. [↗](https://arxiv.org/abs/)
16. Kocmi, Tom, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Ken… [find](https://scholar.google.com/scholar?q=Kocmi%2C+Tom%2C+Ekaterina+Artemova%2C+Eleftherios+Avramidis%2C+Rachel+Bawden%2C+Ondr%CC%8Cej+Bojar%2C+Konstantin+Dranch%2C+Anton+Dvorkovich%2C+Sergey+Dukanov%2C+Mark+Fishel%2C+Markus+Freitag%2C+Thamme+Gowda%2C+Roman+Grundkiewicz%2C) [our card](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
17. Li, Xiang Lisa and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Zong, Chengqing, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference… [find](https://scholar.google.com/scholar?q=Li%2C+Xiang+Lisa+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Zong%2C+Chengqing%2C+Fei+Xia%2C+Wenjie+Li%2C+and+Roberto+Navigli%2C+editors%2C+Proceedings+of+the+59th+Annual+)
18. Liu, Xiao, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Muresan, Smaranda, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Associati… [find](https://scholar.google.com/scholar?q=Liu%2C+Xiao%2C+Kaixuan+Ji%2C+Yicheng+Fu%2C+Weng+Tam%2C+Zhengxiao+Du%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+2022%2E+P%2Dtuning%3A+Prompt+tuning+can+be+comparable+to+fine%2Dtuning+across+scales+and+tasks%2E+In+Muresan%2C+Smaranda%2C+Pres) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/)
19. Lommel, Arle, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 12:455–463. [find](https://scholar.google.com/scholar?q=Lommel%2C+Arle%2C+Aljoscha+Burchardt%2C+and+Hans+Uszkoreit%2E+2014%2E+Multidimensional+quality+metrics+%28mqm%29%3A+A+framework+for+declaring+and+describing+translation+quality+metrics%2E+Traduma%CC%80tica%3A+tecnologies+de+l)
20. Lu, Qingyu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. MQM-APE: Toward high-quality error annotation predictors with automatic post-editing in LLM translation evaluators. In Rambow, Owen, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert… [find](https://scholar.google.com/scholar?q=Lu%2C+Qingyu%2C+Liang+Ding%2C+Kanjian+Zhang%2C+Jinxia+Zhang%2C+and+Dacheng+Tao%2E+2025%2E+MQM%2DAPE%3A+Toward+high%2Dquality+error+annotation+predictors+with+automatic+post%2Dediting+in+LLM+translation+evaluators%2E+In+Rambo) [our card](https://md.research.elamur.ai/free/en/papers/lu-2025-mqm-ape-high-quality-error/)
21. Mizrahi, Moran, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. [find](https://scholar.google.com/scholar?q=Mizrahi%2C+Moran%2C+Guy+Kaplan%2C+Dan+Malkin%2C+Rotem+Dror%2C+Dafna+Shahaf%2C+and+Gabriel+Stanovsky%2E+2024%2E+State+of+what+art%3F+a+call+for+multi%2Dprompt+LLM+evaluation%2E+Transactions+of+the+Association+for+Computatio)
22. Moslem, Yasmin, Gianfranco Romani, Mahdi Molaei, John D. Kelleher, Rejwanul Haque, and Andy Way. 2023. Domain terminology integration into machine translation: Leveraging large language models. In Koehn, Philipp, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conferen… [find](https://scholar.google.com/scholar?q=Moslem%2C+Yasmin%2C+Gianfranco+Romani%2C+Mahdi+Molaei%2C+John+D%2E+Kelleher%2C+Rejwanul+Haque%2C+and+Andy+Way%2E+2023%2E+Domain+terminology+integration+into+machine+translation%3A+Leveraging+large+language+models%2E+In+Koe)
23. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John H… [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+James+Cross%2C+Onur+C%CC%A7elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+)
24. Popović, Maja. 2015. chrf: character n-gram f-score for automatic mt evaluation. In WMT@EMNLP. [find](https://scholar.google.com/scholar?q=Popovic%CC%81%2C+Maja%2E+2015%2E+chrf%3A+character+n%2Dgram+f%2Dscore+for+automatic+mt+evaluation%2E+In+WMT%40EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
25. Post, Matt. 2018. A call for clarity in reporting bleu scores. https://arxiv.org/abs/ 1804.08771. [↗](https://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
26. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+2685%E2%80%932702%2C+Online%2C+November%2E+Association+for+Computational+Linguistics%2E)
27. Sahoo, Pranab, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2025. A systematic survey of prompt engineering in large language models: Techniques and applications. https://arxiv.org/abs/2402.07927. [↗](https://arxiv.org/abs/2402.07927) [our card](https://md.research.elamur.ai/free/en/papers/sahoo-2024-systematic-survey-prompt-engineering-large/)
28. Sclar, Melanie, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. https://arxiv.org/abs/2310.11324. [↗](https://arxiv.org/abs/2310.11324)
29. Sellam, Thibault, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Jurafsky, Dan, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online,… [find](https://scholar.google.com/scholar?q=Sellam%2C+Thibault%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Jurafsky%2C+Dan%2C+Joyce+Chai%2C+Natalie+Schluter%2C+and+Joel+Tetreault%2C+editors%2C+Proceedings+of)
30. Shi, Weijia, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. 2022. Toward human readable prompt tuning: Kubrick’s the shining is a good movie, and a good prompt too? https://arxiv.org/abs/2212.10539. [↗](https://arxiv.org/abs/2212.10539)
31. Shin, Taylor, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Auto-Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Webber, Bonnie, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Method… [find](https://scholar.google.com/scholar?q=Shin%2C+Taylor%2C+Yasaman+Razeghi%2C+Robert+L%2E+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+2020%2E+Auto%2DPrompt%3A+Eliciting+Knowledge+from+Language+Models+with+Automatically+Generated+Prompts%2E+In+Webber%2C+Bonnie%2C+)
32. Voronov, Anton, Lena Wolf, and Max Ryabinin. 2024. Mind your format: Towards consistent evaluation of in-context learning improvements. https://arxiv.org/abs/2401.06766. [↗](https://arxiv.org/abs/2401.06766)
33. White, Jules, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpta prompt pattern catalog to enhance prompt engineering with chatgpt. [https://arxiv.…](https://arxiv.…) [↗](https://arxiv.…) [our card](https://md.research.elamur.ai/free/en/papers/white-2023-prompt-pattern-catalog-enhance-prompt/)
34. Pryzant, Reid, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with ”gradient descent” and beam search. https://arxiv.org/abs/ 2305.03495. [↗](https://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/)
35. Yan, Jianhao, Pingchuan Yan, Yulong Chen, Jing Li, Xianchao Zhu, and Yue Zhang. 2024. Benchmarking gpt-4 against human translators: A comprehensive evaluation across languages, domains, and expertise levels. https://arxiv.org/abs/ 2411.13775. [↗](https://arxiv.org/abs/)
36. Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Webber, Bonnie, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 [find](https://scholar.google.com/scholar?q=Rei%2C+Ricardo%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Webber%2C+Bonnie%2C+Trevor+Cohn%2C+Yulan+He%2C+and+Yang+Liu%2C+editors%2C+Proceedings+of+the+2020) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
37. Yang, Chengrun, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. https://arxiv.org/abs/2309.03409. [↗](https://arxiv.org/abs/2309.03409) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/)
38. Yuksekgonul, Mert, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic ”differentiation” via text. https://arxiv.org/abs/2406. 07496. [↗](https://arxiv.org/abs/2406)
39. Zhang, Ningyu, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022a. Differentiable prompt makes pre-trained language models better few-shot learners. https://arxiv.org/abs/2108.13161. — [arXiv:2108.13161](https://arxiv.org/abs/2108.13161) [↗](https://arxiv.org/abs/2108.13161)
40. Zhang, Tianjun, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022b. Tempera: Test-time prompting via reinforcement learning. https://arxiv.org/abs/2211. 11890. [↗](https://arxiv.org/abs/2211)
41. Zhou, Yulin, Yiren Zhao, Ilia Shumailov, Robert Mullins, and Yarin Gal. 2023. Revisiting automated prompting: Are we actually doing better? https://arxiv.org/abs/2304.03609. [↗](https://arxiv.org/abs/2304.03609)
42. Zhu, Wenhao, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Duh, Kevin, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Comp… [find](https://scholar.google.com/scholar?q=Zhu%2C+Wenhao%2C+Hongyi+Liu%2C+Qingxiu+Dong%2C+Jingjing+Xu%2C+Shujian+Huang%2C+Lingpeng+Kong%2C+Jiajun+Chen%2C+and+Lei+Li%2E+2024%2E+Multilingual+machine+translation+with+large+language+models%3A+Empirical+results+and+anal) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2024-multilingual-machine-translation-large-language/)