---
title: "Scaling Down to Scale Up — paper by Lialin et al. (2023)"
description: "Language models have grown far faster than single-GPU memory, so full fine-tuning of… 101 references, cited by 61 papers here."
authors: ["Vladislav Lialin", "Vijeta Deshpande", "Xiaowei Yao", "Anna Rumshisky"]
year: 2023
venue: "arXiv:2303.15647"
source: "https://arxiv.org/abs/2303.15647"
doi: "10.48550/arxiv.2303.15647"
arxiv: "2303.15647"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lialin-2023-scaling-down-scale-up-guide/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lialin-2023-scaling-down-scale-up-guide/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

# Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

**Authors:** [Vladislav Lialin](https://md.research.elamur.ai/free/en/authors/vladislav-lialin/), Vijeta Deshpande, Xiaowei Yao and [Anna Rumshisky](https://md.research.elamur.ai/free/en/authors/anna-rumshisky/) · **Year:** 2023 · **Venue:** arXiv:2303.15647 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.15647) · [PDF](https://arxiv.org/pdf/2303.15647) · [DOI 10.48550/arxiv.2303.15647](https://doi.org/10.48550/arxiv.2303.15647) · [arXiv 2303.15647](https://arxiv.org/abs/2303.15647) · [OpenAlex](https://openalex.org/W4361230739)

**Citations:** 67 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/lialin-2023-scaling-down-scale-up-guide/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Language models have grown far faster than single-GPU memory, so full fine-tuning of multibillion-parameter models is out of reach for most practitioners. In-context learning is an imperfect substitute because of context limits, unstable results, and costly inference. Lialin et al. survey parameter-efficient fine-tuning (PEFT), in which only a small set of existing or newly added weights is trained. They review more than 50 papers from 2019 to mid-2024 and describe about 30 methods with pseudo-code. These are organized into a taxonomy of addition-based methods (Adapters, Soft Prompts, others), selection-based methods, reparametrization-based methods, and hybrids. The authors also run a large head-to-head benchmark that spans five datasets and models at three scales (0.7B, 3B, 11B), measuring task quality, GPU memory, and throughput. When hyperparameter search is limited and training lasts only a few epochs, methods previously reported to beat LoRA often fail to do so. Plain Layer Norm tuning performs surprisingly well. The paper is a practical guide to choosing PEFT methods and argues that the field needs standardized benchmarks.

## Key points
- **Scope:** 50+ papers and roughly 30 methods are discussed in depth. A category-level comparison (Table 6) covers storage, RAM, speed, inference overhead, and support in HuggingFace PEFT and AdapterHub.
- **Experiments:** The intro reports 14 methods plus variants, while the abstract says 15. They are tested on five datasets with 0.7B, 3B, and 11B models. Code is at github.com/guitaricet/peft_comparison.
- **Results:**
  - LoRA remains a strong baseline.
  - Hybrid methods are highly sensitive to hyperparameters.
  - Kronecker-based reparametrizations save no memory compared with matrix-product variants, but they can speed up training and inference.
- **Reporting gap:** Advertised trainable-parameter counts often differ from actual ones, which makes Prompt Tuning and hybrid methods unexpectedly expensive.
- **Memory rationale:** With Adam, each trainable parameter also needs storage for its gradient and two moment estimates, so training can take 12–20× the memory of the weights alone. PEFT shrinks this overhead. Optimal PEFT learning rates are typically much higher than those for full fine-tuning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 70 works · 39,977 citations
- [University of Massachusetts Lowell](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-lowell/) — University, Lowell, United States · 8 works · 3,758 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao and Anna Rumshisky. 2023. Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning. arXiv:2303.15647.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lialin2023scaling,
  title = {{Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning}},
  author = {Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao and Anna Rumshisky},
  year = {2023},
  doi = {10.48550/arxiv.2303.15647},
  eprint = {2303.15647},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.15647}
}
```

- [← previous Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://md.research.elamur.ai/free/en/papers/deng-2023-prompting-evaluating-large-language-models/)
- [next → Sequence Parallelism: Long Sequence Training from System Perspective](https://md.research.elamur.ai/free/en/papers/li-2023-sequence-parallelism-long-sequence-training/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

101 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). In Annual Meeting of the Association for Computational Linguistics, 2020.
2. Mohammad AkbarTajari, Sara Rajaee, and Mohammad Taher Pilehvar. An empirical study on the transferability of transformer modules in parameter-efficient fine-tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Languag… [find](https://scholar.google.com/scholar?q=Mohammad+AkbarTajari%2C+Sara+Rajaee%2C+and+Mohammad+Taher+Pilehvar%2E+An+empirical+study+on+the+transferability+of+transformer+modules+in+parameter%2Defficient+fine%2Dtuning%2E+In+Yoav+Goldberg%2C+Zornitsa+Kozareva)
3. Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1796, Dublin, Ireland, May 2022. Association for Computa… [find](https://scholar.google.com/scholar?q=Alan+Ansell%2C+Edoardo+Ponti%2C+Anna+Korhonen%2C+and+Ivan+Vuli%C4%87%2E+Composable+sparse+fine%2Dtuning+for+cross%2Dlingual+transfer%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Ling)
4. Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, 2018. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Rong+Ge%2C+Behnam+Neyshabur%2C+and+Yi+Zhang%2E+Stronger+generalization+bounds+for+deep+nets+via+a+compression+approach%2E+In+International+Conference+on+Machine+Learning%2C+2018%2E)
5. Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). ArXiv, abs/1607.06450, 2016.
6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
7. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). ArXiv, abs/2106.10199, 2021.
8. Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. [In-context learning with long-context models: An in-depth exploration](https://md.research.elamur.ai/free/en/papers/bertsch-2025-context-learning-long-context-models/). arXiv preprint arXiv:2405.00200, 2024.
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
10. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper/](https://proceedings.neurips.cc/paper/) 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [↗](https://proceedings.neurips.cc/paper/)
11. Jin Cao, Chandan Prakash, and Wael Hamza. Attention fusion: a light yet efficient late fusion mechanism for task adaptation in nlu. In NAACL-HLT, 2022. [find](https://scholar.google.com/scholar?q=Jin+Cao%2C+Chandan+Prakash%2C+and+Wael+Hamza%2E+Attention+fusion%3A+a+light+yet+efficient+late+fusion+mechanism+for+task+adaptation+in+nlu%2E+In+NAACL%2DHLT%2C+2022%2E)
12. Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.07967, 2023. — [arXiv:2306.07967](https://arxiv.org/abs/2306.07967) [find](https://scholar.google.com/scholar?q=Arnav+Chavan%2C+Zhuang+Liu%2C+Deepak+Gupta%2C+Eric+Xing%2C+and+Zhiqiang+Shen%2E+One%2Dfor%2Dall%3A+Generalized+lora+for+parameter%2Defficient+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2306%2E07967%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E07967%5D%28htt)
13. Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alexander J. Smola, and Diyi Yang. Parameter-efficient fine-tuning design spaces. ArXiv, abs/2301.01821, 2023. [find](https://scholar.google.com/scholar?q=Jiaao+Chen%2C+Aston+Zhang%2C+Xingjian+Shi%2C+Mu+Li%2C+Alexander+J%2E+Smola%2C+and+Diyi+Yang%2E+Parameter%2Defficient+fine%2Dtuning+design+spaces%2E+ArXiv%2C+abs%2F2301%2E01821%2C+2023%2E)
14. David Chiang, Alexander M. Rush, and Boaz Barak. Named tensor notation. ArXiv, abs/2102.13196, 2021. — [arXiv:2102.13196](https://arxiv.org/abs/2102.13196) [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Alexander+M%2E+Rush%2C+and+Boaz+Barak%2E+Named+tensor+notation%2E+ArXiv%2C+abs%2F2102%2E13196%2C+2021%2E+%E2%80%94+%5BarXiv%3A2102%2E13196%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E13196%29)
15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran,… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
16. Tim Dettmers and Luke Zettlemoyer. [The case for 4-bit precision: k-bit inference scaling laws](https://md.research.elamur.ai/free/en/papers/dettmers-2022-case-4-bit-precision-k/). ArXiv, abs/2212.09720, 2022.
17. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E)
18. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2C+2023%2E)
19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
20. Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficien… — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yujia+Qin%2C+Guang+Yang%2C+Fu+Wei%2C+Zonghan+Yang%2C+Yusheng+Su%2C+Shengding+Hu%2C+Yulin+Chen%2C+Chi%2DMin+Chan%2C+Weize+Chen%2C+Jing+Yi%2C+Weilin+Zhao%2C+Xiaozhi+Wang%2C+Zhiyuan+Liu%2C+Haitao+Zheng%2C+Jianfei+Chen%2C+Yan)
21. Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 3… [find](https://scholar.google.com/scholar?q=Jeff+Donahue%2C+Yangqing+Jia%2C+Oriol+Vinyals%2C+Judy+Hoffman%2C+Ning+Zhang%2C+Eric+Tzeng%2C+and+Trevor+Darrell%2E+Decaf%3A+A+deep+convolutional+activation+feature+for+generic+visual+recognition%2E+In+Eric+P%2E+Xing+and+)
22. Ali Edalati, Marzieh S. Tahaei, Ivan Kobyzev, V. Nia, James J. Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. ArXiv, abs/2212.10650, 2022. — [arXiv:2212.10650](https://arxiv.org/abs/2212.10650) [find](https://scholar.google.com/scholar?q=Ali+Edalati%2C+Marzieh+S%2E+Tahaei%2C+Ivan+Kobyzev%2C+V%2E+Nia%2C+James+J%2E+Clark%2C+and+Mehdi+Rezagholizadeh%2E+Krona%3A+Parameter+efficient+tuning+with+kronecker+adapter%2E+ArXiv%2C+abs%2F2212%2E10650%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E106)
23. William Fedus, Barret Zoph, and Noam M. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. ArXiv, abs/2101.03961, 2021. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+M%2E+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+ArXiv%2C+abs%2F2101%2E03961%2C+2021%2E)
24. Cheng Fu, Hanxian Huang, Xinyun Chen, Yuandong Tian, and Jishen Zhao. Learn-to-share: A hardware-friendly transfer learning framework exploiting computation and parameter sharing. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 13… [find](https://scholar.google.com/scholar?q=Cheng+Fu%2C+Hanxian+Huang%2C+Xinyun+Chen%2C+Yuandong+Tian%2C+and+Jishen+Zhao%2E+Learn%2Dto%2Dshare%3A+A+hardware%2Dfriendly+transfer+learning+framework+exploiting+computation+and+parameter+sharing%2E+In+Marina+Meila+and+)
25. Mozhdeh Gheini, Xiang Ren, and Jonathan May. Cross-attention is all you need: Adapting pretrained transformers for machine translation. In Conference on Empirical Methods in Natural Language Processing, 2021. [find](https://scholar.google.com/scholar?q=Mozhdeh+Gheini%2C+Xiang+Ren%2C+and+Jonathan+May%2E+Cross%2Dattention+is+all+you+need%3A+Adapting+pretrained+transformers+for+machine+translation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
26. Demi Guo, Alexander M. Rush, and Yoon Kim. [Parameter-efficient transfer learning with diff pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/). In Annual Meeting of the Association for Computational Linguistics, 2020.
27. Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023, 2023. [find](https://scholar.google.com/scholar?q=Han+Guo%2C+Philip+Greengard%2C+Eric+P+Xing%2C+and+Yoon+Kim%2E+Lq%2Dlora%3A+Low%2Drank+plus+quantized+matrix+decomposition+for+efficient+language+model+finetuning%2E+arXiv+preprint+arXiv%3A2311%2E12023%2C+2023%2E)
28. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1737–… [find](https://scholar.google.com/scholar?q=Suyog+Gupta%2C+Ankur+Agrawal%2C+Kailash+Gopalakrishnan%2C+and+Pritish+Narayanan%2E+Deep+learning+with+limited+numerical+precision%2E+In+Francis+Bach+and+David+Blei%2C+editors%2C+Proceedings+of+the+32nd+Internationa)
29. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. In Annual Meeting of the Association for Computational Linguistics, 2021. [find](https://scholar.google.com/scholar?q=Karen+Hambardzumyan%2C+Hrant+Khachatrian%2C+and+Jonathan+May%2E+Warp%3A+Word%2Dlevel+adversarial+reprogramming%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2021%2E)
30. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In International Conference on Learning Representations, 2022a. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=0RDcd5Axok. [↗](https://openreview.net/forum?)
31. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10.1109/cvpr.2016.90. URL http://dx.doi.org/10.1109/cvpr. 2016.90. [↗](http://dx.doi.org/10.1109/cvpr)
32. Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. SparseAdapter: An easy approach for improving the parameter-efficiency of adapters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2184–2190, Abu Dhabi, United Arab Emirates, December 2022b. Association… [find](https://scholar.google.com/scholar?q=Shwai+He%2C+Liang+Ding%2C+Daize+Dong%2C+Jeremy+Zhang%2C+and+Dacheng+Tao%2E+SparseAdapter%3A+An+easy+approach+for+improving+the+parameter%2Defficiency+of+adapters%2E+In+Findings+of+the+Association+for+Computational+Li)
33. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=d7KBjmI3GmQ. [↗](https://openreview.net/forum?)
34. Connor Holmes, Minjia Zhang, Yuxiong He, and Bo Wu. Nxmtransformer: Semi-structured sparsification for natural language understanding via admm. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages… [find](https://scholar.google.com/scholar?q=Connor+Holmes%2C+Minjia+Zhang%2C+Yuxiong+He%2C+and+Bo+Wu%2E+Nxmtransformer%3A+Semi%2Dstructured+sparsification+for+natural+language+understanding+via+admm%2E+In+M%2E+Ranzato%2C+A%2E+Beygelzimer%2C+Y%2E+Dauphin%2C+P%2ES%2E+Liang%2C+a)
35. Curran Associates, Inc., 2021. URL [https://proceedings.neurips.cc/paper/2021/file/](https://proceedings.neurips.cc/paper/2021/file/) 0e4f5cc9f4f3f7f1651a6b9f9214e5b1-Paper.pdf. [↗](https://proceedings.neurips.cc/paper/2021/file/)
36. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, 2019. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+Intern)
37. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9). [↗](https://openreview.net/forum?id=nZeVKeeFYf9)
38. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv:1809.04281, 2018. — [arXiv:1809.04281](https://arxiv.org/abs/1809.04281) [find](https://scholar.google.com/scholar?q=Cheng%2DZhi+Anna+Huang%2C+Ashish+Vaswani%2C+Jakob+Uszkoreit%2C+Noam+Shazeer%2C+Ian+Simon%2C+Curtis+Hawthorne%2C+Andrew+M+Dai%2C+Matthew+D+Hoffman%2C+Monica+Dinculescu%2C+and+Douglas+Eck%2E+Music+transformer%2E+arXiv+preprint)
39. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. [Lorahub: Efficient cross-task generalization via dynamic lora composition](https://md.research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/), 2023.
40. Hyesung Jeon, Yulhwa Kim, and Jae-joon Kim. L4q: Parameter efficient quantization-aware training on large language models via lora-wise lsq. arXiv preprint arXiv:2402.04902, 2024. — [arXiv:2402.04902](https://arxiv.org/abs/2402.04902) [find](https://scholar.google.com/scholar?q=Hyesung+Jeon%2C+Yulhwa+Kim%2C+and+Jae%2Djoon+Kim%2E+L4q%3A+Parameter+efficient+quantization%2Daware+training+on+large+language+models+via+lora%2Dwise+lsq%2E+arXiv+preprint+arXiv%3A2402%2E04902%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E04902%5D)
41. Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1022–1035. Curran Assoc… [find](https://scholar.google.com/scholar?q=Rabeeh+Karimi+Mahabadi%2C+James+Henderson%2C+and+Sebastian+Ruder%2E+Compacter%3A+Efficient+low%2Drank+hypercomplex+adapter+layers%2E+In+M%2E+Ranzato%2C+A%2E+Beygelzimer%2C+Y%2E+Dauphin%2C+P%2ES%2E+Liang%2C+and+J%2E+Wortman+Vaughan%2C+)
42. Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. On the computational complexity of self-attention. In International Conference on Algorithmic Learning Theory, pages 597–619. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Feyza+Duman+Keles%2C+Pruthuvi+Mahesakya+Wijewardena%2C+and+Chinmay+Hegde%2E+On+the+computational+complexity+of+self%2Dattention%2E+In+International+Conference+on+Algorithmic+Learning+Theory%2C+pages+597%E2%80%93619%2E+PMLR)
43. Mikhail Khrushchev, Ruslan Vasilev, Alexey Petrov, and Nikolay Zinov. YaLM 100B, 6 2022. URL [https://github.com/yandex/YaLM-100B](https://github.com/yandex/YaLM-100B). [↗](https://github.com/yandex/YaLM-100B)
44. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
45. Quoc V. Le, Tamás Sarlós, and Alex Smola. Fastfood: Approximate kernel expansions in loglinear time. ArXiv, abs/1408.3060, 2013. — [arXiv:1408.3060](https://arxiv.org/abs/1408.3060) [find](https://scholar.google.com/scholar?q=Quoc+V%2E+Le%2C+Tam%C3%A1s+Sarl%C3%B3s%2C+and+Alex+Smola%2E+Fastfood%3A+Approximate+kernel+expansions+in+loglinear+time%2E+ArXiv%2C+abs%2F1408%2E3060%2C+2013%2E+%E2%80%94+%5BarXiv%3A1408%2E3060%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1408%2E3060%29)
46. Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. URL [https://proceedings.neurips.cc/paper/1989/file/](https://proceedings.neurips.cc/paper/1989/file/) 6c9882bbac1c7093bd25041881277658-Paper.pdf. [↗](https://proceedings.neurips.cc/paper/1989/file/)
47. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. ArXiv, abs/2104.08691, 2021. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+ArXiv%2C+abs%2F2104%2E08691%2C+2021%2E)
48. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461, 2019.
49. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Chunyuan+Li%2C+Heerad+Farkhoor%2C+Rosanne+Liu%2C+and+Jason+Yosinski%2E+Measuring+the+intrinsic+dimension+of+objective+landscapes%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E)
50. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597,… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Inte)
51. Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023. — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) [find](https://scholar.google.com/scholar?q=Yixiao+Li%2C+Yifan+Yu%2C+Chen+Liang%2C+Pengcheng+He%2C+Nikos+Karampatziakis%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+Loftq%3A+Lora%2Dfine%2Dtuning%2Daware+quantization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E08659%2C+2)
52. Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Stack more layers differently: High-rank training through low-rank updates. arXiv preprint arXiv:2307.[05695, 2023](https://md.research.elamur.ai/free/en/papers/lialin-2023-relora-high-rank-training-through/).
53. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
54. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). ArXiv, abs/2205.05638, 2022.
55. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+arXiv+preprint+arXiv%3A2402%2E09353%2C+2024)
56. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. [Gpt understands, too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/). ArXiv, abs/2103.10385, 2021.
57. Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? arXiv preprint arXiv:2309.01809, 2023. [find](https://scholar.google.com/scholar?q=Sheng+Lu%2C+Irina+Bigoulaeva%2C+Rachneet+Sachdeva%2C+Harish+Tayyar+Madabushi%2C+and+Iryna+Gurevych%2E+Are+emergent+abilities+in+large+language+models+just+in%2Dcontext+learning%3F+arXiv+preprint+arXiv%3A2309%2E01809%2C+2)
58. Wesley J. Maddox, Gregory Benton, and Andrew Gordon Wilson. Rethinking parameter counting: Effective dimensionality revisted. arXiv preprint arXiv:2003.02139, 2020. — [arXiv:2003.02139](https://arxiv.org/abs/2003.02139) [find](https://scholar.google.com/scholar?q=Wesley+J%2E+Maddox%2C+Gregory+Benton%2C+and+Andrew+Gordon+Wilson%2E+Rethinking+parameter+counting%3A+Effective+dimensionality+revisted%2E+arXiv+preprint+arXiv%3A2003%2E02139%2C+2020%2E+%E2%80%94+%5BarXiv%3A2003%2E02139%5D%28https%3A%2F%2Farxiv%2E)
59. Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. arXiv preprint arXiv:2210.05643, 2022. — [arXiv:2210.05643](https://arxiv.org/abs/2210.05643) [find](https://scholar.google.com/scholar?q=Sadhika+Malladi%2C+Alexander+Wettig%2C+Dingli+Yu%2C+Danqi+Chen%2C+and+Sanjeev+Arora%2E+A+kernel%2Dbased+view+of+language+model+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2210%2E05643%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E05643%5D%28https%3A%2F%2Farxi)
60. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft), 2022. [↗](https://github.com/huggingface/peft)
61. Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen tau Yih, and Madian Khabsa. [Unipelt: A unified framework for parameter-efficient language model tuning](https://md.research.elamur.ai/free/en/papers/mao-2022-unipelt-unified-framework-parameter-efficient/). ArXiv, abs/2110.07577, 2021.
62. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. ArXiv, [Adapterfusion: Non-destructive task composition for transfer learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/). abs/2005.00247, 2020a.
63. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. [Adapterhub: A framework for adapting transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020b. doi…
64. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulic, and E. Ponti. Modular deep learning. ArXiv, abs/2302.11529, 2023. — [arXiv:2302.11529](https://arxiv.org/abs/2302.11529) [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Sebastian+Ruder%2C+Ivan+Vulic%2C+and+E%2E+Ponti%2E+Modular+deep+learning%2E+ArXiv%2C+abs%2F2302%2E11529%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E11529%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E11529%29)
65. Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vuli, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: A unified library for parameter-efficient and modular transfer learning. In Yansong Feng and Els Lefever, editors, Proceedings of th… [find](https://scholar.google.com/scholar?q=Clifton+Poth%2C+Hannah+Sterz%2C+Indraneil+Paul%2C+Sukannya+Purkayastha%2C+Leon+Engl%C3%A4nder%2C+Timo+Imhof%2C+Ivan+Vuli%2C+Sebastian+Ruder%2C+Iryna+Gurevych%2C+and+Jonas+Pfeiffer%2E+Adapters%3A+A+unified+library+for+parameter%2D)
66. Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno. Accurate lora-finetuning quantization of llms via information retention. arXiv preprint arXiv:2402.05445, 2024. — [arXiv:2402.05445](https://arxiv.org/abs/2402.05445) [find](https://scholar.google.com/scholar?q=Haotong+Qin%2C+Xudong+Ma%2C+Xingyu+Zheng%2C+Xiaoyang+Li%2C+Yang+Zhang%2C+Shouda+Liu%2C+Jie+Luo%2C+Xianglong+Liu%2C+and+Michele+Magno%2E+Accurate+lora%2Dfinetuning+quantization+of+llms+via+information+retention%2E+arXiv+pre)
67. Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. Exploring universal intrinsic task subspace via prompt tuning. 2021. [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Xiaozhi+Wang%2C+Yusheng+Su%2C+Yankai+Lin%2C+Ning+Ding%2C+Jing+Yi%2C+Weize+Chen%2C+Zhiyuan+Liu%2C+Juanzi+Li%2C+Lei+Hou%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+Exploring+universal+intrinsic+task+subspace+via+pr)
68. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
69. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
70. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018. [doi: 10.18653/v1/p18-2124](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). URL http://dx.doi.org/10.18653/v1/P18-2124. [↗](http://dx.doi.org/10.18653/v1/P18-2124)
71. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. [Learning multiple visual domains with residual adapters](https://md.research.elamur.ai/free/en/papers/rebuffi-2017-learning-multiple-visual-domains-residual/). In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates…
72. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi-domain deep neural networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018. [find](https://scholar.google.com/scholar?q=Sylvestre%2DAlvise+Rebuffi%2C+Hakan+Bilen%2C+and+Andrea+Vedaldi%2E+Efficient+parametrization+of+multi%2Ddomain+deep+neural+networks%2E+2018+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+81)
73. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866, 2020. doi: 10.1162/tacl_a_00349. URL https://aclanthology.org/2020. tacl-1.54. [↗](https://aclanthology.org/2020)
74. Tim Salimans and Diederik P. Kingma. Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 901–909, Red Hook, NY, USA, 2016. Curran Associates Inc.… [find](https://scholar.google.com/scholar?q=Tim+Salimans+and+Diederik+P%2E+Kingma%2E+Weight+normalization%3A+a+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+In+Proceedings+of+the+30th+International+Conference+on+Neural+Inf)
75. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chan… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
76. Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana Ili’c, Daniel Hesslow, Roman Castagn’e, Alexandra Sasha Luccioni, Franccois Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Rose Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot,… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Elizabeth%2DJane+Pavlick%2C+Suzana+Ili%E2%80%99c%2C+Daniel+Hesslow%2C+Roman+Castagn%E2%80%99e%2C+Alexandra+Sasha+Luccioni%2C+Franccois+Yvon%2C+Matthias+Gall%C3%A9%2C+Jonathan+Tow%2C+Alexander+M)
77. Timo Schick and Hinrich Schütze. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 [Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online, June 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). A…
78. Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada, July 2017. Association for… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+L)
79. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL [https://openreview.net/forum?id=B1ckMDqlg](https://openreview.net/forum?id=B1ckMDqlg). [↗](https://openreview.net/forum?id=B1ckMDqlg)
80. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. ArXiv, abs/1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+ArXiv%2C+abs%2F19)
81. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
82. Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Peng Li, Juan-Zi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability of prompt tuning for natural language understanding. ArXiv, abs/2111.06719, 2021. — [arXiv:2111.06719](https://arxiv.org/abs/2111.06719) [find](https://scholar.google.com/scholar?q=Yusheng+Su%2C+Xiaozhi+Wang%2C+Yujia+Qin%2C+Chi%2DMin+Chan%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Juan%2DZi+Li%2C+Lei+Hou%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+On+transferability+of+prompt+tuning+for+natural+language+understand)
83. Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 24193– 24205. Curran Associates, Inc., 2021. URL… [find](https://scholar.google.com/scholar?q=Yi%2DLin+Sung%2C+Varun+Nair%2C+and+Colin+A+Raffel%2E+Training+neural+networks+with+fixed+sparse+masks%2E+In+M%2E+Ranzato%2C+A%2E+Beygelzimer%2C+Y%2E+Dauphin%2C+P%2ES%2E+Liang%2C+and+J%2E+Wortman+Vaughan%2C+editors%2C+Advances+in+Neura)
84. Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. ArXiv, abs/2206.06522, 2022. — [arXiv:2206.06522](https://arxiv.org/abs/2206.06522) [find](https://scholar.google.com/scholar?q=Yi%2DLin+Sung%2C+Jaemin+Cho%2C+and+Mohit+Bansal%2E+Lst%3A+Ladder+side%2Dtuning+for+parameter+and+memory+efficient+transfer+learning%2E+ArXiv%2C+abs%2F2206%2E06522%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E06522%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E06)
85. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
86. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008, 2017.
87. Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Matthew Cer. [Spot: Better frozen model adaptation through soft prompt transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/). In Annual Meeting of the Association for Computational Linguistics, 2021.
88. Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James J. Clark, Brett H. Meyer, and Warren J. Gross. Efficient fine-tuning of bert models on the edge. 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1838–1842, 2022. [find](https://scholar.google.com/scholar?q=Danilo+Vucetic%2C+Mohammadreza+Tayaranian%2C+Maryam+Ziaeefard%2C+James+J%2E+Clark%2C+Brett+H%2E+Meyer%2C+and+Warren+J%2E+Gross%2E+Efficient+fine%2Dtuning+of+bert+models+on+the+edge%2E+2022+IEEE+International+Symposium+on+C)
89. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461, 2018.
90. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). arXiv preprint arXiv:1905.00537, 2019.
91. Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. ArXiv, abs/2205.12410, 2022. — [arXiv:2205.12410](https://arxiv.org/abs/2205.12410) [find](https://scholar.google.com/scholar?q=Yaqing+Wang%2C+Subhabrata+Mukherjee%2C+Xiaodong+Liu%2C+Jing+Gao%2C+Ahmed+Hassan+Awadallah%2C+and+Jianfeng+Gao%2E+Adamix%3A+Mixture%2Dof%2Dadapter+for+parameter%2Defficient+tuning+of+large+language+models%2E+ArXiv%2C+abs%2F2205)
92. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). p…
93. Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. arXiv preprint arXiv:2304.06648, 2023. — [arXiv:2304.06648](https://arxiv.org/abs/2304.06648) [find](https://scholar.google.com/scholar?q=Enze+Xie%2C+Lewei+Yao%2C+Han+Shi%2C+Zhili+Liu%2C+Daquan+Zhou%2C+Zhaoqiang+Liu%2C+Jiawei+Li%2C+and+Zhenguo+Li%2E+Difffit%3A+Unlocking+transferability+of+large+diffusion+models+via+simple+parameter%2Defficient+fine%2Dtuning%2E)
94. Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717, 2023. — [arXiv:2309.14717](https://arxiv.org/abs/2309.14717) [find](https://scholar.google.com/scholar?q=Yuhui+Xu%2C+Lingxi+Xie%2C+Xiaotao+Gu%2C+Xin+Chen%2C+Heng+Chang%2C+Hengheng+Zhang%2C+Zhensu+Chen%2C+Xiaopeng+Zhang%2C+and+Qi+Tian%2E+Qa%2Dlora%3A+Quantization%2Daware+low%2Drank+adaptation+of+large+language+models%2E+arXiv+prepri)
95. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/), 2022.
96. Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Aston+Zhang%2C+Yi+Tay%2C+Shuai+Zhang%2C+Alvin+Chan%2C+Anh+Tuan+Luu%2C+Siu+Cheung+Hui%2C+and+Jie+Fu%2E+Beyond+fully%2Dconnected+layers+with+quaternions%3A+Parameterization+of+hypercomplex+multiplications+with+1%2Fn+parame)
97. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.[10512, 2023a](https://md.research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/).
98. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained tr… — [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/)
99. Zhen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, and Songfang Huang. Towards adaptive prefix tuning for parameter-efficient language model fine-tuning. arXiv preprint arXiv:2305.15212, 2023b. [find](https://scholar.google.com/scholar?q=Zhen%2DRu+Zhang%2C+Chuanqi+Tan%2C+Haiyang+Xu%2C+Chengyu+Wang%2C+Jun+Huang%2C+and+Songfang+Huang%2E+Towards+adaptive+prefix+tuning+for+parameter%2Defficient+language+model+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2305%2E15212%2C)
100. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024. [find](https://scholar.google.com/scholar?q=Jiawei+Zhao%2C+Zhenyu+Zhang%2C+Beidi+Chen%2C+Zhangyang+Wang%2C+Anima+Anandkumar%2C+and+Yuandong+Tian%2E+Galore%3A+Memory%2Defficient+llm+training+by+gradient+low%2Drank+projection%2E+arXiv+preprint+arXiv%3A2403%2E03507%2C+2024)
101. Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. Counter-interference adapter for multilingual machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2812–2823, Punta Cana, Dominican Republic, November 2021. Association for Computatio… [find](https://scholar.google.com/scholar?q=Yaoming+Zhu%2C+Jiangtao+Feng%2C+Chengqi+Zhao%2C+Mingxuan+Wang%2C+and+Lei+Li%2E+Counter%2Dinterference+adapter+for+multilingual+machine+translation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EM)

*Updated 24 October 2026*