--- title: "Parameter-efficient fine-tuning of large-scale — Ding 2023" description: "Updating every weight of ever-larger pre-trained language models (PLMs) is becoming too costly in compute and storage, because each downstream task needs…" authors: ["Ning Ding", "Yujia Qin", "Guang You Yang", "Fuchao Wei", "Zonghan Yang", "Yusheng Su", "Shengding Hu", "Yulin Chen", "Chi-Min Chan", "Weize Chen", "Jing Yi", "Weilin Zhao", "Xiaozhi Wang", "Zhiyuan Liu", "Hai-Tao Zheng", "Jianfei Chen", "Yang Liu", "Jie Tang", "Juanzi Li", "Maosong Sun"] year: 2023 venue: "Nature Machine Intelligence" source: "https://doi.org/10.1038/s42256-023-00626-4" doi: "10.1038/s42256-023-00626-4" licence: "CC BY 4.0 (publisher, via OpenAlex)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ding-2023-parameter-efficient-fine-tuning-large/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ding-2023-parameter-efficient-fine-tuning-large/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Parameter-efficient fine-tuning of large-scale pre-trained language models # Parameter-efficient fine-tuning of large-scale pre-trained language models **Authors:** [Ning Ding](https://md.research.elamur.ai/free/en/authors/ning-ding/), [Yujia Qin](https://md.research.elamur.ai/free/en/authors/yujia-qin/), Guang You Yang, Fuchao Wei, Zonghan Yang, [Yusheng Su](https://md.research.elamur.ai/free/en/authors/yusheng-su/), [Shengding Hu](https://md.research.elamur.ai/free/en/authors/shengding-hu/), [Yulin Chen](https://md.research.elamur.ai/free/en/authors/yulin-chen/), [Chi-Min Chan](https://md.research.elamur.ai/free/en/authors/chi-min-chan/), [Weize Chen](https://md.research.elamur.ai/free/en/authors/weize-chen/), Jing Yi, [Weilin Zhao](https://md.research.elamur.ai/free/en/authors/weilin-zhao/) et al. (20 authors) · **Year:** 2023 · **Venue:** Nature Machine Intelligence · **Licence:** [CC BY 4.0 (publisher, via OpenAlex)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://doi.org/10.1038/s42256-023-00626-4) · [PDF](https://www.nature.com/articles/s42256-023-00626-4.pdf) · [DOI 10.1038/s42256-023-00626-4](https://doi.org/10.1038/s42256-023-00626-4) · [OpenAlex](https://openalex.org/W4322766882) **Citations:** 1167 citations (openalex-snapshot · 2026-10-14) · [157](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Updating every weight of ever-larger pre-trained language models (PLMs) is becoming too costly in compute and storage, because each downstream task needs its own full model copy. The authors group the many lightweight alternatives (Adapter tuning, BitFit, LoRA, prefix-tuning, prompt-tuning and others) under one name, "delta-tuning": a small set of parameters, either existing or newly inserted, is trained while the frozen backbone stays untouched. They formalise the problem and sort methods into addition-based, specification-based and reparameterization-based families. They offer two theoretical lenses. One treats delta-tuning as optimisation inside a low-dimensional subspace. The other treats the delta parameters as optimal controllers that steer hidden-state propagation from layer to layer. Experiments cover more than 100 NLP tasks drawn from CrossFit and GLUE. Delta-tuning comes close to full fine-tuning, no single method wins everywhere, and mixing methods usually helps. With large enough backbones, even randomly chosen tunable parameters become competitive. The work lowers the entry cost of billion-parameter models and clarifies why few-parameter adaptation succeeds. ## Key points - A manual survey of 1,200 papers from ACL, EMNLP and NAACL (2020–2022) found that only about 0.5–4% actually ran PLMs with over 1 billion parameters in their experiments. - Fully fine-tuning GPT-3 would touch roughly 175,255 million parameters. LoRA-style low-rank matrices reduce the trainable portion to about 37.7 million. - Delta-tuning usually converges more slowly than full fine-tuning. Convergence depends more on the delta structure than on the count of tunable parameters, and it improves as model size grows. - Intrinsic prompt-tuning tunes just 250 free parameters in a shared subspace learned from 100 tasks. It recovers 97% of full prompt-tuning performance on seen tasks and 83% on 20 unseen tasks. - Code is released as the OpenDelta toolkit (GitHub thunlp/OpenDelta, also archived on Zenodo). Datasets come from INK-USC/CrossFit and Hugging Face GLUE. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 157 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 151 not yet. - [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/) - [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/) - [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://md.research.elamur.ai/free/en/papers/kmainasi-2025-large-language-models-predict-outcome/) - [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/) - [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ding2023parameter, title = {{Parameter-efficient fine-tuning of large-scale pre-trained language models}}, author = {Ding, Ning and Qin, Yujia and Yang, Guang You and Wei, Fuchao and Yang, Zonghan and Su, Yusheng and Hu, Shengding and Chen, Yulin and Chan, Chi-Min and Chen, Weize and Yi, Jing and Zhao, Weilin and Wang, Xiaozhi and Liu, Zhiyuan and Zheng, Hai-Tao and Chen, Jianfei and Liu, Yang and Tang, Jie and Li, Juanzi and Sun, Maosong}, booktitle = {Nature Machine Intelligence}, year = {2023}, doi = {10.1038/s42256-023-00626-4}, url = {https://doi.org/10.1038/s42256-023-00626-4} } ``` - [← previous ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation](https://md.research.elamur.ai/free/en/papers/huang-2023-paraamr-large-scale-syntactically-diverse/) - [next → Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://md.research.elamur.ai/free/en/papers/krishna-2023-paraphrasing-evades-detectors-ai-generated/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. 2. 3. 4. 5. 6. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015). Hochreiter, S. & Schmidhuber, J. ürgen. Long short-term memory. Neural Comput. 9, 1735–1780 (1997). Bengio, Y., Ducharme, R. & Vincent, P. A neural probabilistic language model. In Advances in Neural Info… [find](https://scholar.google.com/scholar?q=1%2E+2%2E+3%2E+4%2E+5%2E+6%2E+LeCun%2C+Y%2E%2C+Bengio%2C+Y%2E+%26+Hinton%2C+G%2E+Deep+learning%2E+Nature+521%2C+436%E2%80%93444+%282015%29%2E+Hochreiter%2C+S%2E+%26+Schmidhuber%2C+J%2E+%C3%BCrgen%2E+Long+short%2Dterm+memory%2E+Neural+Comput%2E+9%2C+1735%E2%80%931780+%281997%29%2E+Beng) 2. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog. [https://d4mucfpksywv.cloudfront.net/](https://d4mucfpksywv.cloudfront.net/) better-language-models/language-models.pdf (2019). 8. Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21… [↗](https://d4mucfpksywv.cloudfront.net/) 3. Author contributions N.D., Y.Q. and Z.L. initiated and organized the research. N.D. drafted the abstract, the main text and Section Methods. S.H., X.W., W.Z. and Y.Q. added contents to Section Methods. F.W., Z.Y., N.D., Y.Q., S.H. and J.C. discussed the scope and content of the theoretical discussio… — [doi:10.1038/s42256-023-00626-4](https://doi.org/10.1038/s42256-023-00626-4) [find](https://scholar.google.com/scholar?q=Author+contributions+N%2ED%2E%2C+Y%2EQ%2E+and+Z%2EL%2E+initiated+and+organized+the+research%2E+N%2ED%2E+drafted+the+abstract%2C+the+main+text+and+Section+Methods%2E+S%2EH%2E%2C+X%2EW%2E%2C+W%2EZ%2E+and+Y%2EQ%2E+added+contents+to+Section+Methods)