---
title: "AdaLoRA: Adaptive Budget Allocation for — Zhang 2023"
description: "LoRA fine-tunes large pre-trained language models by adding low-rank updates to frozen weights."
authors: ["Qingru Zhang", "Minshuo Chen", "Alexander Bukharin", "Nikos Karampatziakis", "Pengcheng He", "Yu Cheng", "Weizhu Chen", "Tuo Zhao"]
year: 2023
venue: "arXiv:2303.10512"
source: "https://arxiv.org/abs/2303.10512"
doi: "10.48550/arxiv.2303.10512"
arxiv: "2303.10512"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

# AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

**Authors:** Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, [Pengcheng He](https://md.research.elamur.ai/free/en/authors/pengcheng-he/), [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/), [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/), [Tuo Zhao](https://md.research.elamur.ai/free/en/authors/tuo-zhao/) · **Year:** 2023 · **Venue:** arXiv:2303.10512 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.10512) · [PDF](https://arxiv.org/pdf/2303.10512) · [DOI 10.48550/arxiv.2303.10512](https://doi.org/10.48550/arxiv.2303.10512) · [arXiv 2303.10512](https://arxiv.org/abs/2303.10512) · [OpenAlex](https://openalex.org/W4330338093)

**Citations:** 37 citations (openalex-snapshot · 2026-10-14) · [133](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
LoRA fine-tunes large pre-trained language models by adding low-rank updates to frozen weights. However, it gives every weight matrix the same rank, even though matrices differ in how much they matter. The authors show this with DeBERTaV3-base on MNLI: FFN matrices and upper layers benefit more from trainable parameters than attention projections or lower layers.

AdaLoRA instead distributes a fixed parameter budget unevenly. Each update is written in an SVD-like form, P Λ Q, and a penalty pushes P and Q toward orthogonality, so no exact SVD is ever computed. Each singular value and its vectors form a triplet that receives a sensitivity-based importance score. Low-scoring triplets have their singular value zeroed, which lowers that matrix's rank but keeps the vectors so the triplet can recover later. A global scheduler begins with a slightly larger budget and shrinks it toward the target.

The method was tested with DeBERTaV3-base on GLUE, SQuAD v1.1 and SQuAD v2.0, and with BART-large on XSum and CNN/DailyMail. AdaLoRA beats adapters and LoRA, with the clearest gains at very small budgets. This makes cheap per-task fine-tuning more effective.

## Key points
- The update is parameterized as P Λ Q, and pruning acts on singular values only. The orthogonality regularizer falls to roughly 0.001 early in training.
- With under 0.1% of full fine-tuning's trainable parameters, AdaLoRA gains about 1.2 F1 on SQuAD2.0 over the strongest baselines.
- Parameter budgets in the experiments range from 0.08% to 0.65%. Baselines include Houlsby/Pfeiffer adapters and LoRA.
- On a single V100, training overhead compared with LoRA is modest:
  - roughly 11% extra time per epoch on MNLI (105 vs 116–117 min);
  - nearly identical GPU memory (about 11.1 GB).
- An ablation shows LoRA itself improves when applied to all weight matrices rather than only W q and W v. Code is released at github.com/QingruZhang/AdaLoRA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

133 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 128 not yet.

- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
- [大模型参数高效微调方法综述: 技术、趋势与挑战](https://md.research.elamur.ai/free/en/papers/tang-2026-parameter-efficient-fine/)
- [基于大语言模型的中文实体链接实证研究](https://md.research.elamur.ai/free/en/papers/xu-2025-empirical-study-chinese/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2023adalora,
  title = {{AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning}},
  author = {Zhang, Qingru and Chen, Minshuo and Bukharin, Alexander and Karampatziakis, Nikos and He, Pengcheng and Cheng, Yu and Chen, Weizhu and Zhao, Tuo},
  year = {2023},
  doi = {10.48550/arxiv.2303.10512},
  eprint = {2303.10512},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.10512}
}
```

- [← previous Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/)
- [next → HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://md.research.elamur.ai/free/en/papers/wang-2023-huatuo-tuning-llama-model-chinese/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

36 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E+Hinton%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
3. Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956–1982, 2010. [find](https://scholar.google.com/scholar?q=Jian%2DFeng+Cai%2C+Emmanuel+J+Cande%CC%80s%2C+and+Zuowei+Shen%2E+A+singular+value+thresholding+algorithm+for+matrix+completion%2E+SIAM+Journal+on+optimization%2C+20%284%29%3A1956%E2%80%931982%2C+2010%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+A)
5. Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020. [find](https://scholar.google.com/scholar?q=Demi+Guo%2C+Alexander+M+Rush%2C+and+Yoon+Kim%2E+Parameter%2Defficient+transfer+learning+with+diff+pruning%2E+arXiv+preprint+arXiv%3A2012%2E07463%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
6. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=0RDcd5Axok](https://openreview.net/forum?id=0RDcd5Axok). [↗](https://openreview.net/forum?id=0RDcd5Axok) [our card](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/)
7. Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021a. [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+Debertav3%3A+Improving+deberta+using+electra%2Dstyle+pre%2Dtraining+with+gradient%2Ddisentangled+embedding+sharing%2E+arXiv+preprint+arXiv%3A2111%2E09543%2C+2021a%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/)
8. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021b. [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Xiaodong+Liu%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+Deberta%3A+Decoding%2Denhanced+bert+with+disentangled+attention%2E+In+International+Conference+on+Learning+Representations%2C+2021b%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)
9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Teaching+machines+to+read+and+comprehend%2E+Advances+in+neural+information+processi) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790–2799. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+Intern)
11. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9). [↗](https://openreview.net/forum?id=nZeVKeeFYf9)
12. Vladimir Koltchinskii, Karim Lounici, and Alexandre B Tsybakov. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics, 39(5):2302–2329, 2011. [find](https://scholar.google.com/scholar?q=Vladimir+Koltchinskii%2C+Karim+Lounici%2C+and+Alexandre+B+Tsybakov%2E+Nuclear%2Dnorm+penalization+and+optimal+rates+for+noisy+low%2Drank+matrix+completion%2E+The+Annals+of+Statistics%2C+39%285%29%3A2302%E2%80%932329%2C+2011%2E)
13. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational L… [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp%2E+)
14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+language+ge) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
15. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Chengqing+Zong%2C+Fei+Xia%2C+Wenjie+Li%2C+and+Roberto+Navigli+%28eds%2E%29%2C+Proceedings+of+the+59th+Annual+Meeting+of)
16. Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compression to improving generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and… [find](https://scholar.google.com/scholar?q=Chen+Liang%2C+Simiao+Zuo%2C+Minshuo+Chen%2C+Haoming+Jiang%2C+Xiaodong+Liu%2C+Pengcheng+He%2C+Tuo+Zhao%2C+and+Weizhu+Chen%2E+Super+tickets+in+pre%2Dtrained+language+models%3A+From+model+compression+to+improving+generaliza)
17. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pp%2E+74%E2%80%9381%2C+2004%2E)
18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p)
19. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E)
20. Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11264–11272. Computer Vision Foundation / IEEE, 2019. doi: 1… — [doi:10.1109/cvpr.2019.01152](https://doi.org/10.1109/cvpr.2019.01152) [find](https://scholar.google.com/scholar?q=Pavlo+Molchanov%2C+Arun+Mallya%2C+Stephen+Tyree%2C+Iuri+Frosio%2C+and+Jan+Kautz%2E+Importance+estimation+for+neural+network+pruning%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+CVPR+2019%2C+Lon)
21. Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B+Cohen%2C+and+Mirella+Lapata%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+arXiv+preprint+arXiv%3A1808%2E08745%2C+201)
22. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+Alban+Desmaison%2C+Andreas+Ko%CC%88pf%2C+Edward+Yang%2C+Zachary+D) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
23. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Aishwarya+Kamath%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Kyunghyun+Cho%2C+and+Iryna+Gurevych%2E+Adapterfusion%3A+Non%2Ddestructive+task+composition+for+transfer+learning%2E+arXiv+preprint+arXiv%3A2005%2E00247%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
24. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10): 1872–1897, 2020. [find](https://scholar.google.com/scholar?q=Xipeng+Qiu%2C+Tianxiang+Sun%2C+Yige+Xu%2C+Yunfan+Shao%2C+Ning+Dai%2C+and+Xuanjing+Huang%2E+Pre%2Dtrained+models+for+natural+language+processing%3A+A+survey%2E+Science+China+Technological+Sciences%2C+63%2810%29%3A+1872%E2%80%931897%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+Peter+J+Liu%2C+et+al%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+tr)
27. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, 2016. Association for Computational Linguistics. do… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+La) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
28. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, Melbourne, Australia, 2018. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
29. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=Sylvestre%2DAlvise+Rebuffi%2C+Hakan+Bilen%2C+and+Andrea+Vedaldi%2E+Learning+multiple+visual+domains+with+residual+adapters%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/rebuffi-2017-learning-multiple-visual-domains-residual/)
30. Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. 2020. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Thomas+Wolf%2C+and+Alexander+M%2E+Rush%2E+Movement+pruning%3A+Adaptive+sparsity+by+fine%2Dtuning%2E+2020%2E)
31. Kim-Chuan Toh and Sangwoon Yun. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of optimization, 6(615-640):15, 2010. [find](https://scholar.google.com/scholar?q=Kim%2DChuan+Toh+and+Sangwoon+Yun%2E+An+accelerated+proximal+gradient+algorithm+for+nuclear+norm+regularized+linear+least+squares+problems%2E+Pacific+Journal+of+optimization%2C+6%28615%2D640%29%3A15%2C+2010%2E)
32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+7th+International+Con) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv preprint, abs/1910.03771, 2019. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+et+al%2E+Huggingface%E2%80%99s+transformers%3A+State%2Dof%2Dthe%2Dart+n) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
34. Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020. [find](https://scholar.google.com/scholar?q=Greg+Yang+and+Edward+J+Hu%2E+Feature+learning+in+infinite%2Dwidth+neural+networks%2E+arXiv+preprint+arXiv%3A2011%2E14522%2C+2020%2E)
35. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021. [find](https://scholar.google.com/scholar?q=Elad+Ben+Zaken%2C+Shauli+Ravfogel%2C+and+Yoav+Goldberg%2E+Bitfit%3A+Simple+parameter%2Defficient+fine%2Dtuning+for+transformer%2Dbased+masked+language%2Dmodels%2E+arXiv+preprint+arXiv%3A2106%2E10199%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/)
36. Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pp. 26809–26823. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Qingru+Zhang%2C+Simiao+Zuo%2C+Chen+Liang%2C+Alexander+Bukharin%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+Platon%3A+Pruning+large+transformer+models+with+upper+confidence+bound+of+weight+importance%2E+In+Interna)