---
title: "Full Parameter Fine-tuning for Large Language Models with"
description: "This paper proposes LOMO (LOw-Memory Optimization), which makes full fine-tuning feasible on… 37 references, cited by 47 papers here."
authors: ["Kai Lv", "Yuqing Yang", "Tengxiao Liu", "Qipeng Guo", "Xipeng Qiu"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.445/"
doi: "10.18653/v1/2024.acl-long.445"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lv-2024-full-parameter-fine-tuning-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lv-2024-full-parameter-fine-tuning-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / Full Parameter Fine-tuning for Large Language Models with Limited Resources

# Full Parameter Fine-tuning for Large Language Models with Limited Resources

**Authors:** [Kai Lv](https://md.research.elamur.ai/free/en/authors/kai-lv/), [Yuqing Yang](https://md.research.elamur.ai/free/en/authors/yuqing-yang/), Tengxiao Liu, [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/) and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.445/) · [PDF](https://aclanthology.org/2024.acl-long.445.pdf) · [DOI 10.18653/v1/2024.acl-long.445](https://doi.org/10.18653/v1/2024.acl-long.445) · [OpenAlex](https://openalex.org/W4402671897)

**Citations:** 96 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/lv-2024-full-parameter-fine-tuning-large/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Full parameter fine-tuning of large language models usually requires expensive hardware such as multiple 80GB GPUs. Parameter-efficient methods like LoRA lower that cost, but they update only a small part of the model. This paper proposes LOMO (LOw-Memory Optimization), which makes full fine-tuning feasible on consumer GPUs.

The authors argue that plain SGD is adequate for fine-tuning LLMs. Their reasoning is that loss surfaces in such models are likely smooth, a local optimum is acceptable, and saddle points lie far from pre-trained weights. Switching to SGD removes Adam's optimizer states entirely. LOMO then updates each parameter as soon as its gradient is computed during backpropagation and frees that gradient immediately, so only one gradient tensor needs to be held at a time. Gradient normalization, loss scaling and selected full-precision operations keep mixed-precision training stable.

Combined with existing memory-saving techniques, this allows a 65B LLaMA model to be fully fine-tuned on a single machine with eight RTX 3,090 cards. On SuperGLUE tasks, LOMO generally beats LoRA. The approach therefore lowers the hardware barrier for smaller research groups.

## Key points
- Peak memory falls to 10.8% of a standard DeepSpeed setup. The gradient memory cost becomes O(1), bounded by the largest single gradient tensor.
- Full fine-tuning of LLaMA-65B ran on 8×RTX 3090 (24GB each).
- The evaluation used six SuperGLUE datasets (RTE, BoolQ, WSC, WIC, MultiRC, COPA), each with 1,000 training examples. Average scores for Zero-shot / LoRA / LOMO were:
  - 13B: 57.2 / 81.4 / 84.2
  - 30B: 58.4 / 86.4 / 87.1
  - 65B: 61.3 / 89.0 / 89.9
- Combining LoRA with LOMO on LLaMA-13B (BoolQ, MultiRC) improved results further, which suggests the two methods are complementary.
- Limitations: gradient normalization currently needs an extra backward pass, and throughput was not measured on A100 GPUs. Code is available at github.com/OpenLMLab/LOMO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo and Xipeng Qiu. 2024. Full Parameter Fine-tuning for Large Language Models with Limited Resources. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lv2024full,
  title = {{Full Parameter Fine-tuning for Large Language Models with Limited Resources}},
  author = {Kai Lv, Yuqing Yang, Tengxiao Liu, Qipeng Guo and Xipeng Qiu},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.445},
  url = {https://doi.org/10.18653/v1/2024.acl-long.445}
}
```

- [← previous Executable Code Actions Elicit Better LLM Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/)
- [next → Generative Verifiers: Reward Modeling as Next-Token Prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+2016%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2E)
2. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 201… [find](https://scholar.google.com/scholar?q=Zhao+Chen%2C+Vijay+Badrinarayanan%2C+Chen%2DYu+Lee%2C+and+Andrew+Rabinovich%2E+2018%2E+Gradnorm%3A+Gradient+normalization+for+adaptive+loss+balancing+in+deep+multitask+networks%2E+In+Proceedings+of+the+35th+Internati)
3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
4. (Long and Short Papers), pages 2924–2936. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=%28Long+and+Short+Papers%29%2C+pages+2924%E2%80%932936%2E+Association+for+Computational+Linguistics%2E)
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005,… [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges%2C+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classificati)
6. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. [8-bit optimizers via block-wise quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
7. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2022. Delta tuning: A comprehensive study of param… — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yujia+Qin%2C+Guang+Yang%2C+Fuchao+Wei%2C+Zonghan+Yang%2C+Yusheng+Su%2C+Shengding+Hu%2C+Yulin+Chen%2C+Chi%2DMin+Chan%2C+Weize+Chen%2C+Jing+Yi%2C+Weilin+Zhao%2C+Xiaozhi+Wang%2C+Zhiyuan+Liu%2C+Hai%2DTao+Zheng%2C+Jianfei+Chen)
8. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. [Visualizing and understanding the effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/hao-2019-visualizing-understanding-effectiveness-bert/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, Ch…
9. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
10. Kenji Kawaguchi, Jiaoyang Huang, and Leslie Pack Kaelbling. 2019. Every local minimum value is the global minimum value of induced model in nonconvex machine learning. Neural Comput., 31(12):2293– 2323. [find](https://scholar.google.com/scholar?q=Kenji+Kawaguchi%2C+Jiaoyang+Huang%2C+and+Leslie+Pack+Kaelbling%2E+2019%2E+Every+local+minimum+value+is+the+global+minimum+value+of+induced+model+in+nonconvex+machine+learning%2E+Neural+Comput%2E%2C+31%2812%29%3A2293%E2%80%93+232)
11. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+3rd+International+Conference+on+Learning+Representations%2C+ICLR+2015%2C+San+Diego%2C+CA%2C+USA%2C+May+7%2D9%2C+2015%2C+Conference)
13. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Principles of Knowledge Representation and Reasoning: Proceedings of the Thirteenth International Conference, KR 2012, Rome, Italy, June 10-14, 2012. AAAI Press. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Principles+of+Knowledge+Representation+and+Reasoning%3A+Proceedings+of+the+Thirteenth+International+Confe)
14. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
15. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
16. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. CoRR, abs/2305.17333. [find](https://scholar.google.com/scholar?q=Sadhika+Malladi%2C+Tianyu+Gao%2C+Eshaan+Nichani%2C+Alex+Damian%2C+Jason+D%2E+Lee%2C+Danqi+Chen%2C+and+Sanjeev+Arora%2E+2023%2E+Fine%2Dtuning+language+models+with+just+forward+passes%2E+CoRR%2C+abs%2F2305%2E17333%2E)
17. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+and+Hao+Wu%2E+2018%2E+Mixed+precision+trai)
18. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. Proceedings of the Internatio… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+2021%2E+Efficient+)
19. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E)
20. Mohammad Taher Pilehvar and José Camacho-Collados. 2019. [Wic: the word-in-context dataset for evaluating context-sensitive meaning representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL…
21. Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. 2020. Training large neural networks with constant memory using a new execution algorithm. arXiv preprint arXiv:2002.05645. [find](https://scholar.google.com/scholar?q=Bharadwaj+Pudipeddi%2C+Maral+Mesmakhosroshahi%2C+Jinwen+Xi%2C+and+Sujeeth+Bharadwaj%2E+2020%2E+Training+large+neural+networks+with+constant+memory+using+a+new+execution+algorithm%2E+arXiv+preprint+arXiv%3A2002%2E0564)
22. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1– 16. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+SC20%3A+International+Conference+for+High+Performance+Comput)
23. Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+Jeff+Rasley%2C+Shaden+Smith%2C+and+Yuxiong+He%2E+2021%2E+Zero%2Dinfinity%3A+Breaking+the+gpu+memory+wall+for+extreme+scale+deep+learning%2E+Proceedings+of+the+International+Conf)
24. Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li. 2021a. Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 598–611. [find](https://scholar.google.com/scholar?q=Jie+Ren%2C+Jiaolin+Luo%2C+Kai+Wu%2C+Minjia+Zhang%2C+Hyeran+Jeon%2C+and+Dong+Li%2E+2021a%2E+Sentinel%3A+Efficient+tensor+migration+and+allocation+on+heterogeneous+memory+systems+for+deep+learning%2E+In+2021+IEEE+Interna)
25. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021b. [Zero-offload: Democratizing billion-scale model training](https://md.research.elamur.ai/free/en/papers/ren-2021-zero-offload-democratizing-billion-scale/). USENIX Annual Technical Conference, pages 551– 564.
26. Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W. Keckler. 2016. vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1–13. [find](https://scholar.google.com/scholar?q=Minsoo+Rhu%2C+Natalia+Gimelshein%2C+Jason+Clemons%2C+Arslan+Zulfiqar%2C+and+Stephen+W%2E+Keckler%2E+2016%2E+vdnn%3A+Virtualized+deep+neural+networks+for+scalable%2C+memory%2Defficient+neural+network+design%2E+In+2016+49th+)
27. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Symposium, Technical Report SS-11-06, Stanford, California, USA, Ma… [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S%2E+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+Logical+Formalizations+of+Commonsense+Reasoning%2C+)
28. Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. CoRR, abs/1609.04747. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+2016%2E+An+overview+of+gradient+descent+optimization+algorithms%2E+CoRR%2C+abs%2F1609%2E04747%2E)
29. Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhao. 2020a. A survey of optimization methods from a machine learning perspective. IEEE Trans. Cybern., 50(8):3668–3681. [find](https://scholar.google.com/scholar?q=Shiliang+Sun%2C+Zehui+Cao%2C+Han+Zhu%2C+and+Jing+Zhao%2E+2020a%2E+A+survey+of+optimization+methods+from+a+machine+learning+perspective%2E+IEEE+Trans%2E+Cybern%2E%2C+50%288%29%3A3668%E2%80%933681%2E)
30. Xianghui Sun, Yunjie Ji, Baochang Ma, and Xiangang Li. 2023. A comparative study between full-parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model. CoRR, abs/2304.08109. — [arXiv:2304.08109](https://arxiv.org/abs/2304.08109) [find](https://scholar.google.com/scholar?q=Xianghui+Sun%2C+Yunjie+Ji%2C+Baochang+Ma%2C+and+Xiangang+Li%2E+2023%2E+A+comparative+study+between+full%2Dparameter+and+lora%2Dbased+fine%2Dtuning+on+chinese+instruction+data+for+instruction+following+large+language+)
31. Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. 2020b. Ultra-low precision 4-bit training of deep neural networks. In Advances in Neural Information Processing Systems 33:… [find](https://scholar.google.com/scholar?q=Xiao+Sun%2C+Naigang+Wang%2C+Chia%2DYu+Chen%2C+Jiamin+Ni%2C+Ankur+Agrawal%2C+Xiaodong+Cui%2C+Swagath+Venkataramani%2C+Kaoutar+El+Maghraoui%2C+Vijayalakshmi+Srinivasan%2C+and+Kailash+Gopalakrishnan%2E+2020b%2E+Ultra%2Dlow+precis)
32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, ab…
33. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural I…
34. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Song, Zenglin Xu, and Tim Kraska. 2018. Superneurons: dynamic gpu memory management for training deep neural networks. ACM SIGPLAN Notices, 53:41–53. [find](https://scholar.google.com/scholar?q=Linnan+Wang%2C+Jinmian+Ye%2C+Yiyang+Zhao%2C+Wei+Wu%2C+Ang+Li%2C+Shuaiwen+Song%2C+Zenglin+Xu%2C+and+Tim+Kraska%2E+2018%2E+Superneurons%3A+dynamic+gpu+memory+management+for+training+deep+neural+networks%2E+ACM+SIGPLAN+Notice)
35. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/).
36. Trans. Mach. Learn. Res., 2022. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2022%2E)
37. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient LLM training by gradient low-rank projection. CoRR, abs/2403.03507. [find](https://scholar.google.com/scholar?q=Jiawei+Zhao%2C+Zhenyu+Zhang%2C+Beidi+Chen%2C+Zhangyang+Wang%2C+Anima+Anandkumar%2C+and+Yuandong+Tian%2E+2024%2E+Galore%3A+Memory%2Defficient+LLM+training+by+gradient+low%2Drank+projection%2E+CoRR%2C+abs%2F2403%2E03507%2E)

*Updated 24 October 2026*