---
title: "High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning"
description: "The authors present DistZO2, a framework that spreads ZO fine-tuning across a CPU plus several GPUs. 26 references."
authors: ["Liangyu Wang", "Huanyi Xie", "Di Wang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.2128/"
doi: "10.18653/v1/2026.findings-acl.2128"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-high-throughput-memory-efficient-zeroth/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-high-throughput-memory-efficient-zeroth/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/) / High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing

# High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing

**Authors:** Liangyu Wang, Huanyi Xie and [Di Wang](https://md.research.elamur.ai/free/en/authors/di-wang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.2128/) · [PDF](https://aclanthology.org/2026.findings-acl.2128.pdf) · [DOI 10.18653/v1/2026.findings-acl.2128](https://doi.org/10.18653/v1/2026.findings-acl.2128) · [OpenAlex](https://openalex.org/W7166826864)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Zeroth-order (ZO) optimization fine-tunes LLMs using only forward passes, which saves memory. The earlier ZO2 framework fits models as large as OPT-175B on one GPU by keeping parameters in CPU memory and streaming transformer blocks to the device. However, it runs on a single device and is slow. The authors present DistZO2, a framework that spreads ZO fine-tuning across a CPU plus several GPUs.

DistZO2 has three parts:
- **Perturbation Parallelism (PertP):** the positively and negatively perturbed forward passes run on separate GPUs. A shared random seed keeps the perturbations identical, and only scalar losses are exchanged.
- **ZO-adapted DDP:** devices synchronize the scalar projected gradient rather than full gradient tensors.
- **2D layout:** the two schemes are combined.

To relieve the PCIe bottleneck, parameter blocks are sliced and redistributed between GPUs over NVLink. On models up to OPT-175B, DistZO2 trains about three times faster than ZO2 while keeping GPU memory low. This makes full-parameter fine-tuning of very large models feasible on modest multi-GPU hardware.

## Key points
- On OPT-175B, DistZO2 reaches roughly a 3× speedup over ZO2, with GPU memory staying below about 20 GB.
- PertP and the ZO-adapted DDP recover the throughput lost to ZO2's CPU offloading, bringing it to the level of MeZO.
- Experiments covered OPT-1.3B through OPT-30B and OPT-175B.
- In the 4-GPU 2D setup, placing PertP inner and DDP outer consistently gave higher throughput than the reverse order.
- NVLink-based block slicing raised upload and offload bandwidth by up to 4.6×.
- Scalability was verified on multi-node clusters of up to 16 GPUs. Pre-training, hybrid ZO/first-order methods and multimodal models are left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [King Abdullah University of Science and Technology](https://md.research.elamur.ai/free/en/universities/king-abdullah-university-of-science-and-technology/) — University, Jeddah, Saudi Arabia · 27 works · 1,084 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 81 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works · [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 19 works

## Cite

Cite as: Liangyu Wang, Huanyi Xie and Di Wang. 2026. High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026high,
  title = {{High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing}},
  author = {Liangyu Wang, Huanyi Xie and Di Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2128},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2128}
}
```

- [← previous High-Order Question Generation in a Multilingual Educational Context](https://md.research.elamur.ai/free/en/papers/ucar-2026-high-order-question-generation-multilingual/)
- [next → HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering](https://md.research.elamur.ai/free/en/papers/shin-2026-hikey-hierarchical-multimodal-retrieval-open/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/)

## References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
2. Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, and 1 others. 2024. [Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation](https://md.research.elamur.ai/free/en/papers/ansel-2024-pytorch-2-faster-machine-learning/). In Proceedings of the 29th ACM…
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
4. HanQin Cai, Yuchen Lou, Daniel McKenzie, and Wotao Yin. 2021. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In International Conference on Machine Learning, pages 1193–1203. PMLR. [find](https://scholar.google.com/scholar?q=HanQin+Cai%2C+Yuchen+Lou%2C+Daniel+McKenzie%2C+and+Wotao+Yin%2E+2021%2E+A+zeroth%2Dorder+block+coordinate+descent+algorithm+for+huge%2Dscale+black%2Dbox+optimization%2E+In+International+Conference+on+Machine+Learning%2C+)
5. Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. 2004. Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007. [find](https://scholar.google.com/scholar?q=Abraham+D+Flaxman%2C+Adam+Tauman+Kalai%2C+and+H+Brendan+McMahan%2E+2004%2E+Online+convex+optimization+in+the+bandit+setting%3A+gradient+descent+without+a+gradient%2E+arXiv+preprint+cs%2F0408007%2E)
6. Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019. Beyond data and model parallelism for deep neural networks. Proceedings of Machine Learning and Systems, 1:1–13. [find](https://scholar.google.com/scholar?q=Zhihao+Jia%2C+Matei+Zaharia%2C+and+Alex+Aiken%2E+2019%2E+Beyond+data+and+model+parallelism+for+deep+neural+networks%2E+Proceedings+of+Machine+Learning+and+Systems%2C+1%3A1%E2%80%9313%2E)
7. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
8. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2E)
9. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and 1 others. 2020. [Pytorch distributed: Experiences on accelerating data parallel training](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/). arXiv preprint arXiv:2006.15704.
10. Liangyu Wang, Junxiao Wang, Jie Ren, Zihang Xiang, David E Keyes, and Di Wang. 2025b. Private training large-scale models with efficient dp-sgd. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Liangyu+Wang%2C+Junxiao+Wang%2C+Jie+Ren%2C+Zihang+Xiang%2C+David+E+Keyes%2C+and+Di+Wang%2E+2025b%2E+Private+training+large%2Dscale+models+with+efficient+dp%2Dsgd%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neural+Informa)
11. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412)
12. Liangyu Wang, Siqi Zhang, Junjie Wang, Yiming Dong, Bo Zheng, Zihan Qiu, Shengkun Tang, Di Wang, Rui Men, and Dayiheng Liu. 2026. Canzona: A unified, asynchronous, and load-balanced framework for distributed matrix-based optimizers. arXiv preprint arXiv:2602.06079. [find](https://scholar.google.com/scholar?q=Liangyu+Wang%2C+Siqi+Zhang%2C+Junjie+Wang%2C+Yiming+Dong%2C+Bo+Zheng%2C+Zihan+Qiu%2C+Shengkun+Tang%2C+Di+Wang%2C+Rui+Men%2C+and+Dayiheng+Liu%2E+2026%2E+Canzona%3A+A+unified%2C+asynchronous%2C+and+load%2Dbalanced+framework+for+dist)
13. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
14. Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075. [find](https://scholar.google.com/scholar?q=Sadhika+Malladi%2C+Tianyu+Gao%2C+Eshaan+Nichani%2C+Alex+Damian%2C+Jason+D+Lee%2C+Danqi+Chen%2C+and+Sanjeev+Arora%2E+2023%2E+Fine%2Dtuning+language+models+with+just+forward+passes%2E+Advances+in+Neural+Information+Process)
15. Yurii Nesterov and Vladimir Spokoiny. 2017. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566. [find](https://scholar.google.com/scholar?q=Yurii+Nesterov+and+Vladimir+Spokoiny%2E+2017%2E+Random+gradient%2Dfree+minimization+of+convex+functions%2E+Foundations+of+Computational+Mathematics%2C+17%282%29%3A527%E2%80%93566%2E)
16. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2018. [Wic: the word-in-context dataset for evaluating context-sensitive meaning representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). arXiv preprint arXiv:1808.09121.
17. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1– 16. IEEE. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Com)
18. Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+2016%2E+An+overview+of+gradient+descent+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1609%2E04747%2E)
19. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
20. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
21. Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, and 1 others. 2024. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592. — [arXiv:2402.11592](https://arxiv.org/abs/2402.11592) [find](https://scholar.google.com/scholar?q=Yihua+Zhang%2C+Pingzhi+Li%2C+Junyuan+Hong%2C+Jiaxiang+Li%2C+Yimeng+Zhang%2C+Wenqing+Zheng%2C+Pin%2DYu+Chen%2C+Jason+D+Lee%2C+Wotao+Yin%2C+Mingyi+Hong%2C+and+1+others%2E+2024%2E+Revisiting+zeroth%2Dorder+optimization+for+memory%2De)
22. Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, and 1 others. 2022. Alpa: Automating inter-and {Intra-Operator} parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and I… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Zhuohan+Li%2C+Hao+Zhang%2C+Yonghao+Zhuang%2C+Zhifeng+Chen%2C+Yanping+Huang%2C+Yida+Wang%2C+Yuanzhong+Xu%2C+Danyang+Zhuo%2C+Eric+P+Xing%2C+and+1+others%2E+2022%2E+Alpa%3A+Automating+inter%2Dand+%7BIntra%2DOperator%7D+p)
23. Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864. [find](https://scholar.google.com/scholar?q=Tim+Salimans%2C+Jonathan+Ho%2C+Xi+Chen%2C+Szymon+Sidor%2C+and+Ilya+Sutskever%2E+2017%2E+Evolution+strategies+as+a+scalable+alternative+to+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1703%2E03864%2E)
24. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+p)
25. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631…
26. Liangyu Wang, Jie Ren, Hang Xu, Junxiao Wang, Huanyi Xie, David E Keyes, and Di Wang. 2025a. Zo2: Scalable zeroth-order fine-tuning for extremely large language models with limited gpu memory. arXiv preprint arXiv:2503.12668. [find](https://scholar.google.com/scholar?q=Liangyu+Wang%2C+Jie+Ren%2C+Hang+Xu%2C+Junxiao+Wang%2C+Huanyi+Xie%2C+David+E+Keyes%2C+and+Di+Wang%2E+2025a%2E+Zo2%3A+Scalable+zeroth%2Dorder+fine%2Dtuning+for+extremely+large+language+models+with+limited+gpu+memory%2E+arXiv+p)

*Updated 1 November 2026*