---
title: "Ultra-Low-Dimensional Prompt Tuning via Random Projection"
description: "Wu, Hao and Mou propose ULPT (Ultra-Low-dimensional Prompt Tuning). 81 references."
authors: ["Zijun Wu", "Yongchang Hao", "Lili Mou"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.59/"
doi: "10.18653/v1/2026.eacl-long.59"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2026-ultra-low-dimensional-prompt-tuning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2026-ultra-low-dimensional-prompt-tuning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) / Ultra-Low-Dimensional Prompt Tuning via Random Projection

# Ultra-Low-Dimensional Prompt Tuning via Random Projection

**Authors:** Zijun Wu, Yongchang Hao and [Lili Mou](https://md.research.elamur.ai/free/en/authors/lili-mou/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.59/) · [PDF](https://aclanthology.org/2026.eacl-long.59.pdf) · [DOI 10.18653/v1/2026.eacl-long.59](https://doi.org/10.18653/v1/2026.eacl-long.59) · [OpenAlex](https://openalex.org/W7140130272)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Prompt tuning adapts a frozen LLM by learning a few input embeddings. Each embedding normally has the model's full hidden size, so the trainable parameter count grows as models get larger. Wu, Hao and Mou propose ULPT (Ultra-Low-dimensional Prompt Tuning). It learns prompt vectors in a very small space, as low as 2 dimensions, and maps them to model size with a random projection matrix that is never trained. Because the matrix can be rebuilt from a random seed, it costs almost nothing to store. Two small learned vectors, a shift and a scale, help align the projected prompts with the model's embedding space. The authors give a convergence analysis and argue theoretically that random projection keeps the pairwise dot-product structure that attention relies on. Across more than 20 tasks on T5, Bloomz and Llama, ULPT matches or beats vanilla prompt tuning while training up to 98% fewer parameters. With a somewhat larger dimension, it also outperforms recent parameter-efficient methods. This makes it attractive for cheaply storing many per-task or per-user customizations.

## Key points

- **Parameter savings:** trainable parameters drop from nr + rd (as in DPT, which also learns the projection) to nr. The only additions are the 2d shift/scale parameters and one random seed.
- **Evaluation:** benchmarks include GLUE, SuperGLUE, MRQA, GSM8K, MBPP and four commonsense reasoning tasks. Code is published at github.com/MANGA-UOFA/ULPT.
- **Prompt length vs. dimension:** under an equal parameter budget, 100 low-dimensional tokens consistently beat fewer full-dimensional tokens. This held on MNLI and Natural Questions with T5-small, T5-base and T5-large, and also for decoder-only models.
- **Freezing P vs. training P (T5-base):** tuning P instead of Z cannot fit a 1.7K-parameter budget, where ULPT scores 81.9 on MNLI. At 27.1K parameters, ULPT still edges ahead: 85.4 vs. 85.3 on MNLI and 72.6 vs. 72.0 on NQ.
- **Ablation:** removing the shift and scale vectors hurts optimization, especially at r = 2. Models with tens to hundreds of billions of parameters were not tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Alberta](https://md.research.elamur.ai/free/en/universities/university-of-alberta/) — University, Edmonton, Canada · 51 works · 5,017 citations
- [Canadian Institute for Advanced Research](https://md.research.elamur.ai/free/en/labs/canadian-institute-for-advanced-research/) — Research institute, Toronto, Canada · 35 works · 19,316 citations
- [Alberta Machine Intelligence Institute](https://md.research.elamur.ai/free/en/labs/alberta-machine-intelligence-institute/) — Research institute, Alberta, Canada · 5 works · 231 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Zijun Wu, Yongchang Hao and Lili Mou. 2026. Ultra-Low-Dimensional Prompt Tuning via Random Projection. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2026ultra,
  title = {{Ultra-Low-Dimensional Prompt Tuning via Random Projection}},
  author = {Zijun Wu, Yongchang Hao and Lili Mou},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.59},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.59}
}
```

- [← previous uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2026-uircis-semeval-2026-task-8/)
- [next → UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization Fingerprint](https://md.research.elamur.ai/free/en/papers/zheng-2026-ummf-protecting-copyright-large-vision/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/)

## References

81 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro…
2. Akari Asai, Mohammadreza Salehi, Matthew Peters, and Hannaneh Hajishirzi. 2022. [ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts](https://md.research.elamur.ai/free/en/papers/asai-2022-attempt-parameter-efficient-multi-task/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6655–6672.
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
4. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. [BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 1–9.
5. Ella Bingham and Heikki Mannila. 2001. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 245–250. [find](https://scholar.google.com/scholar?q=Ella+Bingham+and+Heikki+Mannila%2E+2001%2E+Random+projection+in+dimensionality+reduction%3A+applications+to+image+and+text+data%2E+In+Proceedings+of+the+Seventh+ACM+SIGKDD+International+Conference+on+Knowledg)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
7. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. [SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation, pages 1–14.
8. Joon-Young Choi, Junho Kim, Jun-Hyung Park, Wing-Lam Mok, and SangKeun Lee. 2023. SMoP: Towards efficient and effective prompt tuning with sparse mixture-of-prompts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14306–14316. [find](https://scholar.google.com/scholar?q=Joon%2DYoung+Choi%2C+Junho+Kim%2C+Jun%2DHyung+Park%2C+Wing%2DLam+Mok%2C+and+SangKeun+Lee%2E+2023%2E+SMoP%3A+Towards+efficient+and+effective+prompt+tuning+with+sparse+mixture%2Dof%2Dprompts%2E+In+Proceedings+of+the+2023+Confere)
9. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
11. Sanjoy Dasgupta and Anupam Gupta. 2003. An elementary proof of a theorem of Johnson and Lindenstrauss. Random Structures & Algorithms, 22(1):60–65. [find](https://scholar.google.com/scholar?q=Sanjoy+Dasgupta+and+Anupam+Gupta%2E+2003%2E+An+elementary+proof+of+a+theorem+of+Johnson+and+Lindenstrauss%2E+Random+Structures+%26+Algorithms%2C+22%281%29%3A60%E2%80%9365%2E)
12. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. [The CommitmentBank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/). In Proceedings of Sinn und Bedeutung, pages 107– 124.
13. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. [Transforming question answering datasets into natural language inference datasets](https://md.research.elamur.ai/free/en/papers/demszky-2018-transforming-question-answering-datasets-natural/). arXiv preprint arXiv:1809.02922.
14. William B. Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing.
15. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. [A survey on in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128.
16. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. [SearchQA: A new Q&A dataset augmented with context from a search engine](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/). arXiv preprint arXiv:1704.05179.
17. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. [find](https://scholar.google.com/scholar?q=Adam+Fisch%2C+Alon+Talmor%2C+Robin+Jia%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Danqi+Chen%2E+2019%2E)
18. [MRQA 2019 shared task: Evaluating generalization in reading comprehension](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/). In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13.
19. Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. 2024. Parameter-efficient fine-tuning with discrete Fourier transform. In Proceedings of the 41st International Conference on Machine Learning, pages 14884– 14901. [find](https://scholar.google.com/scholar?q=Ziqi+Gao%2C+Qichao+Wang%2C+Aochuan+Chen%2C+Zijing+Liu%2C+Bingzhe+Wu%2C+Liang+Chen%2C+and+Jia+Li%2E+2024%2E+Parameter%2Defficient+fine%2Dtuning+with+discrete+Fourier+transform%2E+In+Proceedings+of+the+41st+International+Con)
20. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. [The third PASCAL recognizing textual entailment challenge](https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/). In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9.
21. Shouchang Guo, Sonam Damani, and Keng-hao Chang. 2024. LoPT: Low-rank prompt tuning for parameter efficient language models. arXiv preprint arXiv:2406.19486. — [arXiv:2406.19486](https://arxiv.org/abs/2406.19486) [find](https://scholar.google.com/scholar?q=Shouchang+Guo%2C+Sonam+Damani%2C+and+Keng%2Dhao+Chang%2E+2024%2E+LoPT%3A+Low%2Drank+prompt+tuning+for+parameter+efficient+language+models%2E+arXiv+preprint+arXiv%3A2406%2E19486%2E+%E2%80%94+%5BarXiv%3A2406%2E19486%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
22. Yongchang Hao, Yanshuai Cao, and Lili Mou. 2024. Flora: Low-rank adapters are secretly gradient compressors. In Proceedings of the 41st International Conference on Machine Learning, pages 17554– 17571. [find](https://scholar.google.com/scholar?q=Yongchang+Hao%2C+Yanshuai+Cao%2C+and+Lili+Mou%2E+2024%2E+Flora%3A+Low%2Drank+adapters+are+secretly+gradient+compressors%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning%2C+pages+17554%E2%80%93+17571)
23. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. [LoRA+: Efficient low rank adaptation of large models](https://md.research.elamur.ai/free/en/papers/hayou-2024-lora-efficient-low-rank-adaptation/). In Proceedings of the 41st International Conference on Machine Learning, pages 17783–17806.
24. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790–2799. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
25. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
26. Piotr Indyk and Rajeev Motwani. 1998. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing, pages 604–613. [find](https://scholar.google.com/scholar?q=Piotr+Indyk+and+Rajeev+Motwani%2E+1998%2E+Approximate+nearest+neighbors%3A+Towards+removing+the+curse+of+dimensionality%2E+In+Proceedings+of+the+Thirtieth+Annual+ACM+Symposium+on+Theory+of+Computing%2C+pages+60)
27. Hamish Ivison and Matthew Peters. 2022. Hyperde-coders: Instance-specific decoders for multi-task NLP. In Findings of the Association for Computational Linguistics: EMNLP, pages 1715–1730. [find](https://scholar.google.com/scholar?q=Hamish+Ivison+and+Matthew+Peters%2E+2022%2E+Hyperde%2Dcoders%3A+Instance%2Dspecific+decoders+for+multi%2Dtask+NLP%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP%2C+pages+1715%E2%80%931730%2E)
28. Hamed Karimi, Julie Nutini, and Mark Schmidt. 2016. Linear convergence of gradient and proximal-gradient methods under the Polyak-Lojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases, pages 795–811. [find](https://scholar.google.com/scholar?q=Hamed+Karimi%2C+Julie+Nutini%2C+and+Mark+Schmidt%2E+2016%2E+Linear+convergence+of+gradient+and+proximal%2Dgradient+methods+under+the+Polyak%2DLojasiewicz+condition%2E+In+Machine+Learning+and+Knowledge+Discovery+in+)
29. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Preced-dings of the 13th International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+schema+challenge%2E+In+Preced%2Ddings+of+the+13th+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reason)
30. Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. [Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks](https://md.research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf…
31. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4582–4597. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2DTuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
32. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
33. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+32%281%29%2E)
34. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems, pages 22199–22213.
35. Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2024. [VeRA: Vector-based random matrix adaptation](https://md.research.elamur.ai/free/en/papers/kopiczko-2023-vera-vector-based-random-matrix/). In International Conference on Learning Representations.
36. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
37. Pengxiang Lan, Haoyu Xu, Enneng Yang, Yuliang Liang, Guibing Guo, Jianzhe Zhao, and Xingwei Wang. 2025. Efficient and effective prompt tuning via prompt decomposition and compressed outer product. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com… [find](https://scholar.google.com/scholar?q=Pengxiang+Lan%2C+Haoyu+Xu%2C+Enneng+Yang%2C+Yuliang+Liang%2C+Guibing+Guo%2C+Jianzhe+Zhao%2C+and+Xingwei+Wang%2E+2025%2E+Efficient+and+effective+prompt+tuning+via+prompt+decomposition+and+compressed+outer+product%2E+In+)
38. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
39. Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2024. [ReLoRA: High-rank training through low-rank updates](https://md.research.elamur.ai/free/en/papers/lialin-2023-relora-high-rank-training-through/). In International Conference on Learning Representations.
40. Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). In Advances in Neural Information Processing Systems.
41. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022b. [P-Tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 61–68.
42. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2024a. GPT understands, too. AI Open, pages 208–215. [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Yanan+Zheng%2C+Zhengxiao+Du%2C+Ming+Ding%2C+Yujie+Qian%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+2024a%2E+GPT+understands%2C+too%2E+AI+Open%2C+pages+208%E2%80%93215%2E)
43. Xinyue Liu, Harshita Diddee, and Daphne Ippolito. 2024b. Customizing large language model generation style using parameter-efficient finetuning. In Proceedings of the 17th International Natural Language Generation Conference, pages 412–426. [find](https://scholar.google.com/scholar?q=Xinyue+Liu%2C+Harshita+Diddee%2C+and+Daphne+Ippolito%2E+2024b%2E+Customizing+large+language+model+generation+style+using+parameter%2Defficient+finetuning%2E+In+Proceedings+of+the+17th+International+Natural+Langua)
44. Jiří Matoušek. 2008. On variants of the Johnson– Lindenstrauss lemma. Random Structures & Algorithms, 33(2):142–156. [find](https://scholar.google.com/scholar?q=Jir%CC%8C%C3%AD+Matou%C5%A1ek%2E+2008%2E+On+variants+of+the+Johnson%E2%80%93+Lindenstrauss+lemma%2E+Random+Structures+%26+Algorithms%2C+33%282%29%3A142%E2%80%93156%2E)
45. Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. 2020. On the global convergence rates of Softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, pages 6820–6829. [find](https://scholar.google.com/scholar?q=Jincheng+Mei%2C+Chenjun+Xiao%2C+Csaba+Szepesvari%2C+and+Dale+Schuurmans%2E+2020%2E+On+the+global+convergence+rates+of+Softmax+policy+gradient+methods%2E+In+Proceedings+of+the+37th+International+Conference+on+Mach)
46. AI @ Meta. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
47. AI @ Meta. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
48. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
49. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. LISA: Layerwise importance sampling for memory-efficient large language model fine-tuning. In Advances in Neural Information Processing Systems, pages 57018–57049. [find](https://scholar.google.com/scholar?q=Rui+Pan%2C+Xiang+Liu%2C+Shizhe+Diao%2C+Renjie+Pi%2C+Jipeng+Zhang%2C+Chi+Han%2C+and+Tong+Zhang%2E+2024%2E+LISA%3A+Layerwise+importance+sampling+for+memory%2Defficient+large+language+model+fine%2Dtuning%2E+In+Advances+in+Neura)
50. Aleksandar Petrov, Philip Torr, and Adel Bibi. 2024a. Prompting a pretrained transformer can be a universal approximator. In Proceedings of the 41st International Conference on Machine Learning, pages 40523–40550. [find](https://scholar.google.com/scholar?q=Aleksandar+Petrov%2C+Philip+Torr%2C+and+Adel+Bibi%2E+2024a%2E+Prompting+a+pretrained+transformer+can+be+a+universal+approximator%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning%2C+pages)
51. Aleksandar Petrov, Philip Torr, and Adel Bibi. 2024b. When do prompting and prefix-tuning work? A theory of capabilities and limitations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Aleksandar+Petrov%2C+Philip+Torr%2C+and+Adel+Bibi%2E+2024b%2E+When+do+prompting+and+prefix%2Dtuning+work%3F+A+theory+of+capabilities+and+limitations%2E+In+International+Conference+on+Learning+Representations%2E)
52. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. [WiC: The word-in-context dataset for evaluating context-sensitive meaning representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages…
53. Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. 2021. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arXiv:2110.07867. — [arXiv:2110.07867](https://arxiv.org/abs/2110.07867) [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Xiaozhi+Wang%2C+Yusheng+Su%2C+Yankai+Lin%2C+Ning+Ding%2C+Jing+Yi%2C+Weize+Chen%2C+Zhiyuan+Liu%2C+Juanzi+Li%2C+Lei+Hou%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2021%2E+Exploring+universal+intrinsic+task+subspace+)
54. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, pages 1–67.
55. Anastasiia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. 2023. Residual Prompt Tuning: Improving prompt tuning with residual reparameterization. In Findings of the Association for Computational Linguistics: ACL, pages 6740–6757. [find](https://scholar.google.com/scholar?q=Anastasiia+Razdaibiedina%2C+Yuning+Mao%2C+Madian+Khabsa%2C+Mike+Lewis%2C+Rui+Hou%2C+Jimmy+Ba%2C+and+Amjad+Almahairi%2E+2023%2E+Residual+Prompt+Tuning%3A+Improving+prompt+tuning+with+residual+reparameterization%2E+In+Find)
56. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. [AdapterDrop: On the efficiency of adapters in transformers](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930–7946.
57. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, page 99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+WinoGrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+page+99%E2%80%93106%2E)
58. Zhengxiang Shi and Aldo Lipani. 2024. DePT: Decomposed prompt tuning for parameter-efficient fine-tuning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhengxiang+Shi+and+Aldo+Lipani%2E+2024%2E+DePT%3A+Decomposed+prompt+tuning+for+parameter%2Defficient+fine%2Dtuning%2E+In+International+Conference+on+Learning+Representations%2E)
59. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting knowledge from language models with automatically generated prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4222–4235.
60. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–…
61. Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. LST: Ladder side-tuning for parameter and memory efficient transfer learning. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yi%2DLin+Sung%2C+Jaemin+Cho%2C+and+Mohit+Bansal%2E+2022%2E+LST%3A+Ladder+side%2Dtuning+for+parameter+and+memory+efficient+transfer+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
62. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. [NewsQA: A machine comprehension dataset](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/). In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200.
63. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems.
64. Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou’, and Daniel Cer. 2022. [SPoT: Better frozen model adaptation through soft prompt transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 5039–5059.
65. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [SuperGlue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems.
66. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
67. Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhen+Wang%2C+Rameswar+Panda%2C+Leonid+Karlinsky%2C+Rogerio+Feris%2C+Huan+Sun%2C+and+Yoon+Kim%2E+2023%2E+Multitask+prompt+tuning+enables+parameter%2Defficient+transfer+learning%2E+In+The+Eleventh+International+Conferenc)
68. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, pages 625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+pages+625%E2%80%93641%2E)
69. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E)
70. Dai, and Quoc V Le. 2022a. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations.
71. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, pages 24824–24837.
72. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112–1122.
73. Xun Wu, Shaohan Huang, and Furu Wei. 2024a. Mixture of LoRA experts. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xun+Wu%2C+Shaohan+Huang%2C+and+Furu+Wei%2E+2024a%2E+Mixture+of+LoRA+experts%2E+In+International+Conference+on+Learning+Representations%2E)
74. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024b. ReFT: Representation fine-tuning for language models. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zhengxuan+Wu%2C+Aryaman+Arora%2C+Zheng+Wang%2C+Atticus+Geiger%2C+Dan+Jurafsky%2C+Christopher+D+Manning%2C+and+Christopher+Potts%2E+2024b%2E+ReFT%3A+Representation+fine%2Dtuning+for+language+models%2E+In+Advances+in+Neural+)
75. Zijun Wu, Yongkang Wu, and Lili Mou. 2024c. Zero-shot continuous prompt transfer: Generalizing task semantics across language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zijun+Wu%2C+Yongkang+Wu%2C+and+Lili+Mou%2E+2024c%2E+Zero%2Dshot+continuous+prompt+transfer%3A+Generalizing+task+semantics+across+language+models%2E+In+International+Conference+on+Learning+Representations%2E)
76. Yao Xiao, Lu Xu, Jiaxi Li, Wei Lu, and Xiaoli Li. 2023. Decomposed prompt tuning via low-rank reparameterization. In Findings of the Association for Computational Linguistics: EMNLP, pages 13335–13347. [find](https://scholar.google.com/scholar?q=Yao+Xiao%2C+Lu+Xu%2C+Jiaxi+Li%2C+Wei+Lu%2C+and+Xiaoli+Li%2E+2023%2E+Decomposed+prompt+tuning+via+low%2Drank+reparameterization%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP%2C+pages+13335%E2%80%9313347)
77. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
78. Fangcong Yin, Xi Ye, and Greg Durrett. 2024. LoFiT: Localized fine-tuning on LLM representations. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Fangcong+Yin%2C+Xi+Ye%2C+and+Greg+Durrett%2E+2024%2E+LoFiT%3A+Localized+fine%2Dtuning+on+LLM+representations%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
79. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Advances in Neural Information Processing Systems, page 649–657.
80. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. [PAWS: Paraphrase adversaries from word scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1298–1308.
81. Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. 2024. Tuning layernorm in attention: Towards efficient multi-modal LLM finetuning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bingchen+Zhao%2C+Haoqin+Tu%2C+Chen+Wei%2C+Jieru+Mei%2C+and+Cihang+Xie%2E+2024%2E+Tuning+layernorm+in+attention%3A+Towards+efficient+multi%2Dmodal+LLM+finetuning%2E+In+International+Conference+on+Learning+Representation)

*Updated 2 November 2026*