---
title: "Fine-Tuning Language Models with Just Forward Passes (2023)"
description: "They introduce MeZO, an in-place version of ZO-SGD built on the SPSA estimator. 113 references, cited by 87 papers here."
authors: ["Sadhika Malladi", "Tianyu Gao", "Eshaan Nichani", "Alex Damian", "Jason D. Lee", "Danqi Chen", "Sanjeev Arora"]
year: 2023
venue: "arXiv:2305.17333"
source: "https://arxiv.org/abs/2305.17333"
doi: "10.48550/arxiv.2305.17333"
arxiv: "2305.17333"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/malladi-2023-fine-tuning-language-models-just/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/malladi-2023-fine-tuning-language-models-just/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Fine-Tuning Language Models with Just Forward Passes

# Fine-Tuning Language Models with Just Forward Passes

**Authors:** [Sadhika Malladi](https://md.research.elamur.ai/free/en/authors/sadhika-malladi/), [Tianyu Gao](https://md.research.elamur.ai/free/en/authors/tianyu-gao/), Eshaan Nichani, Alex Damian, [Jason D. Lee](https://md.research.elamur.ai/free/en/authors/jason-d-lee/), [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) and [Sanjeev Arora](https://md.research.elamur.ai/free/en/authors/sanjeev-arora/) · **Year:** 2023 · **Venue:** [arXiv:2305.17333](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.17333) · [PDF](https://arxiv.org/pdf/2305.17333) · [DOI 10.48550/arxiv.2305.17333](https://doi.org/10.48550/arxiv.2305.17333) · [arXiv 2305.17333](https://arxiv.org/abs/2305.17333) · [OpenAlex](https://openalex.org/W7133241261)

**Citations:** 66 citations (OpenAlex, 2026-10-14) · [87](https://md.research.elamur.ai/free/en/papers/malladi-2023-fine-tuning-language-models-just/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Fine-tuning large language models with backpropagation is memory-hungry. Activations, gradients and Adam's optimizer state can need roughly an order of magnitude more GPU memory than inference alone. The authors revisit zeroth-order optimization, which estimates gradients from loss differences rather than backward passes. They introduce MeZO, an in-place version of ZO-SGD built on the SPSA estimator. Instead of storing the random perturbation vector, MeZO regenerates it from a saved random seed, so training uses about as much memory as inference.

They test MeZO on masked LMs (RoBERTa-large) and autoregressive LMs (OPT, 350M up to 66B) across classification, multiple-choice and generation tasks. MeZO clearly beats zero-shot, in-context learning and linear probing, and often comes close to full fine-tuning. It works with both full-parameter tuning and PEFT such as LoRA and prefix tuning, and it can optimize non-differentiable metrics directly. A theoretical analysis explains why: with good pre-training and prompts, the slowdown depends on the landscape's local effective rank rather than on the parameter count. This makes tuning very large models feasible on modest hardware.

## Key points
- On one A100 80GB GPU, MeZO can tune a 30B model, while backpropagation with Adam fits only about a 2.7B model. Memory savings reach up to 12×.
- With OPT-13B, MeZO is within 1% of Adam fine-tuning (or better) on 7 of 11 tasks, including SST-2, RTE, BoolQ, SQuAD and DROP. With RoBERTa-large, the gap to standard fine-tuning stays under 5%.
- For a 30B model, the authors' implementation needs about half the GPU-hours of Adam fine-tuning. The main limitation is that MeZO requires many optimization steps.
- A MeZO-tuned 66B checkpoint can be stored in under 0.1MB: one seed plus a 2-byte scalar for each of 20,000 steps. LoRA needs about 38MB and prefix tuning about 12MB.
- Using a single random perturbation per step (n = 1) was the most efficient setting. Momentum and Adam variants, plus layerwise-adaptive versions, are described but brought no efficiency gain in early tests.

## Interesting ideas
- **Regenerate randomness instead of storing it.** Resetting the RNG with a saved seed reproduces the same noise vector whenever it is needed. Any algorithm that applies large random perturbations repeatedly, such as noise injection or randomized search, could reuse this to avoid holding huge tensors in memory.
- **A training run as seed plus scalars.** Because each update is just one number times seed-generated noise, a whole fine-tuning run can be logged and replayed from a tiny record. This could make sharing many task-specific adaptations of one base model very cheap, for example across several Ukrainian NLP tasks.
- **Optimize the metric you actually care about.** Since only loss values are needed, the objective can be accuracy, F1 or another non-differentiable score. This is useful when the evaluation metric differs from cross-entropy, or for preference-style rewards.
- **Pre-training makes the problem low-dimensional.** The argument that a well-pretrained model plus a task prompt yields a landscape with small effective rank explains why cheap, noisy optimizers still work. It suggests designing prompts and pre-training so that downstream adaptation stays easy.
- **Trade memory against time at the granularity you choose.** Perturbing whole weight matrices instead of individual scalars speeds things up at the cost of memory equal to the largest matrix (about 0.86GB for OPT-66B). This is a simple knob for balancing memory and speed on limited hardware.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen and Sanjeev Arora. 2023. Fine-Tuning Language Models with Just Forward Passes. arXiv:2305.17333.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{malladi2023fine,
  title = {{Fine-Tuning Language Models with Just Forward Passes}},
  author = {Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen and Sanjeev Arora},
  year = {2023},
  doi = {10.48550/arxiv.2305.17333},
  eprint = {2305.17333},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.17333}
}
```

- [← previous ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages](https://md.research.elamur.ai/free/en/papers/robinson-2023-chatgpt-mt-competitive-high-but/)
- [next → Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

113 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hany S Abdel-Khalik, Paul D Hovland, Andrew Lyons, Tracy E Stover, and Jean Utke. A low rank approach to automatic differentiation. In Advances in Automatic Differentiation, pages 55–65, 2008. [find](https://scholar.google.com/scholar?q=Hany+S+Abdel%2DKhalik%2C+Paul+D+Hovland%2C+Andrew+Lyons%2C+Tracy+E+Stover%2C+and+Jean+Utke%2E+A+low+rank+approach+to+automatic+differentiation%2E+In+Advances+in+Automatic+Differentiation%2C+pages+55%E2%80%9365%2C+2008%2E)
2. Menachem Adelman, Kfir Levy, Ido Hakimi, and Mark Silberstein. Faster neural network training with approximate tensor operations. Advances in Neural Information Processing Systems, 34:27877–27889, 2021. [find](https://scholar.google.com/scholar?q=Menachem+Adelman%2C+Kfir+Levy%2C+Ido+Hakimi%2C+and+Mark+Silberstein%2E+Faster+neural+network+training+with+approximate+tensor+operations%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A27877%E2%80%9327889%2C+202)
3. Alekh Agarwal, Peter L. Bartlett, Pradeep Ravikumar, and Martin J. Wainwright. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory, 58(5):3235–3249, May 2012. [find](https://scholar.google.com/scholar?q=Alekh+Agarwal%2C+Peter+L%2E+Bartlett%2C+Pradeep+Ravikumar%2C+and+Martin+J%2E+Wainwright%2E+Information%2Dtheoretic+lower+bounds+on+the+oracle+complexity+of+stochastic+convex+optimization%2E+IEEE+Transactions+on+Infor)
4. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin…
5. Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. [Promptsource: An integrated development environment and repository for natural language prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/). arXiv preprint arXiv:2202.01279, 2022.
6. Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order (non)-convex stochastic optimization via conditional gradient and gradient updates. In Advances in Neural Information Processing Systems, volume 31, 2018. [find](https://scholar.google.com/scholar?q=Krishnakumar+Balasubramanian+and+Saeed+Ghadimi%2E+Zeroth%2Dorder+%28non%29%2Dconvex+stochastic+optimization+via+conditional+gradient+and+gradient+updates%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+v)
7. Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, pages 1–42, 2022. [find](https://scholar.google.com/scholar?q=Krishnakumar+Balasubramanian+and+Saeed+Ghadimi%2E+Zeroth%2Dorder+nonconvex+stochastic+optimization%3A+Handling+constraints%2C+high+dimensionality%2C+and+saddle+points%2E+Foundations+of+Computational+Mathematics%2C+)
8. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+Second+PASCA)
9. Atılım Güneş Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation, 2022. [find](https://scholar.google.com/scholar?q=At%C4%B1l%C4%B1m+G%C3%BCnes%CC%A7+Baydin%2C+Barak+A%2E+Pearlmutter%2C+Don+Syme%2C+Frank+Wood%2C+and+Philip+Torr%2E+Gradients+without+backpropagation%2C+2022%2E)
10. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In TAC, 2009. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+and+Danilo+Giampiccolo%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+TAC%2C+2009%2E)
11. Raghu Bollapragada, Richard Byrd, and Jorge Nocedal. Adaptive sampling strategies for stochastic optimization. SIAM Journal on Optimization, 28(4):3312–3343, 2018. [find](https://scholar.google.com/scholar?q=Raghu+Bollapragada%2C+Richard+Byrd%2C+and+Jorge+Nocedal%2E+Adaptive+sampling+strategies+for+stochastic+optimization%2E+SIAM+Journal+on+Optimization%2C+28%284%29%3A3312%E2%80%933343%2C+2018%2E)
12. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, 2015.
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in neural information processing systems, volume 33, pages 1877–1901, 2020.
14. HanQin Cai, Yuchen Lou, Daniel McKenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In International Conference on Machine Learning, pages 1193–1203, 2021. [find](https://scholar.google.com/scholar?q=HanQin+Cai%2C+Yuchen+Lou%2C+Daniel+McKenzie%2C+and+Wotao+Yin%2E+A+zeroth%2Dorder+block+coordinate+descent+algorithm+for+huge%2Dscale+black%2Dbox+optimization%2E+In+International+Conference+on+Machine+Learning%2C+pages+)
15. HanQin Cai, Daniel McKenzie, Wotao Yin, and Zhenliang Zhang. Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2):687–714, 2022. [find](https://scholar.google.com/scholar?q=HanQin+Cai%2C+Daniel+McKenzie%2C+Wotao+Yin%2C+and+Zhenliang+Zhang%2E+Zeroth%2Dorder+regularized+optimization+%28zoro%29%3A+Approximately+sparse+gradients+and+adaptive+sampling%2E+SIAM+Journal+on+Optimization%2C+32%282%29%3A687)
16. Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Clip-tuning: Towards derivative-free prompt learning with a mixture of rewards. arXiv preprint arXiv:2210.12050, 2022. [find](https://scholar.google.com/scholar?q=Yekun+Chai%2C+Shuohuan+Wang%2C+Yu+Sun%2C+Hao+Tian%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+Clip%2Dtuning%3A+Towards+derivative%2Dfree+prompt+learning+with+a+mixture+of+rewards%2E+arXiv+preprint+arXiv%3A2210%2E12050%2C+2022%2E)
17. Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15–26, 2017. [find](https://scholar.google.com/scholar?q=Pin%2DYu+Chen%2C+Huan+Zhang%2C+Yash+Sharma%2C+Jinfeng+Yi%2C+and+Cho%2DJui+Hsieh%2E+Zoo%3A+Zeroth+order+optimization+based+black%2Dbox+attacks+to+deep+neural+networks+without+training+substitute+models%2E+In+Proceedings+o)
18. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
19. Krzysztof M Choromanski and Vikas Sindhwani. On blackbox backpropagation and jacobian sensing. In Advances in Neural Information Processing Systems, volume 30, 2017. [find](https://scholar.google.com/scholar?q=Krzysztof+M+Choromanski+and+Vikas+Sindhwani%2E+On+blackbox+backpropagation+and+jacobian+sensing%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+30%2C+2017%2E)
20. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in neural information processing systems, volume 30, 2017. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Advances+in+neural+information+processing+systems%2C+volume+3)
21. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
22. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, 2005. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+the+First+International+Conference+on+Machine+Learning+Challenges%3A+Evaluating+Predictive+Uncerta)
23. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volum)
24. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. [The commitmentbank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/). In Sinn und Bedeutung, volume 23, pages 107–124, 2019.
25. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. [RLPrompt: Optimizing discrete text prompts with reinforcement learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369–3391, 202…
26. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+GPT3%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
27. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. [8-bit optimizers via block-wise quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/). In International Conference on Learning Representations, 2022.
28. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
29. Shizhe Diao, Xuechun Li, Yong Lin, Zhichao Huang, and Tong Zhang. Black-box prompt learning for pre-trained language models. arXiv preprint arXiv:2201.08531, 2022. — [arXiv:2201.08531](https://arxiv.org/abs/2201.08531) [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Xuechun+Li%2C+Yong+Lin%2C+Zhichao+Huang%2C+and+Tong+Zhang%2E+Black%2Dbox+prompt+learning+for+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2201%2E08531%2C+2022%2E+%E2%80%94+%5BarXiv%3A2201%2E08531%5D%28https%3A%2F%2Farxiv%2Eo)
30. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904, 2022. — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yujia+Qin%2C+Guang+Yang%2C+Fuchao+Wei%2C+Zonghan+Yang%2C+Yusheng+Su%2C+Shengding+Hu%2C+Yulin+Chen%2C+Chi%2DMin+Chan%2C+Weize+Chen%2C+et+al%2E+Delta+tuning%3A+A+comprehensive+study+of+parameter+efficient+methods+fo)
31. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
32. John C. Duchi, Michael I. Jordan, Martin J. Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015. [find](https://scholar.google.com/scholar?q=John+C%2E+Duchi%2C+Michael+I%2E+Jordan%2C+Martin+J%2E+Wainwright%2C+and+Andre+Wibisono%2E+Optimal+rates+for+zero%2Dorder+convex+optimization%3A+The+power+of+two+function+evaluations%2E+IEEE+Transactions+on+Information+Th)
33. FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training, 2021. [find](https://scholar.google.com/scholar?q=FairScale+authors%2E+Fairscale%3A+A+general+purpose+modular+pytorch+library+for+high+performance+and+large+scale+training%2C+2021%2E)
34. Abraham D. Flaxman, Adam Tauman Kalai, and H. Brendan McMahan. Online convex optimization in the bandit setting: Gradient descent without a gradient. In Proceedings of the Sixteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’05, page 385–394, USA, 2005. Society for Industrial and Applie… [find](https://scholar.google.com/scholar?q=Abraham+D%2E+Flaxman%2C+Adam+Tauman+Kalai%2C+and+H%2E+Brendan+McMahan%2E+Online+convex+optimization+in+the+bandit+setting%3A+Gradient+descent+without+a+gradient%2E+In+Proceedings+of+the+Sixteenth+Annual+ACM%2DSIAM+Sy)
35. Tianyu Gao, Adam Fisch, and Danqi Chen. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 38…
36. Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pages 2232–2241, 2019. [find](https://scholar.google.com/scholar?q=Behrooz+Ghorbani%2C+Shankar+Krishnan%2C+and+Ying+Xiao%2E+An+investigation+into+neural+net+optimization+via+hessian+eigenvalue+density%2E+In+International+Conference+on+Machine+Learning%2C+pages+2232%E2%80%932241%2C+2019%2E)
37. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. [The third PASCAL recognizing textual entailment challenge](https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/). In the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, 2007.
38. Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, and Qiuyi Zhang. Gradientless descent: High-dimensional zeroth-order optimization. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Daniel+Golovin%2C+John+Karro%2C+Greg+Kochanski%2C+Chansoo+Lee%2C+Xingyou+Song%2C+and+Qiuyi+Zhang%2E+Gradientless+descent%3A+High%2Ddimensional+zeroth%2Dorder+optimization%2E+In+International+Conference+on+Learning+Repres)
39. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=Priya+Goyal%2C+Piotr+Doll%C3%A1r%2C+Ross+Girshick%2C+Pieter+Noordhuis%2C+Lukasz+Wesolowski%2C+Aapo+Kyrola%2C+Andrew+Tulloch%2C+Yangqing+Jia%2C+and+Kaiming+He%2E+Accurate%2C+large+minibatch+sgd%3A+Training+imagenet+in+1+hour%2E+ar)
40. Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008. [find](https://scholar.google.com/scholar?q=Andreas+Griewank+and+Andrea+Walther%2E+Evaluating+derivatives%3A+principles+and+techniques+of+algorithmic+differentiation%2E+SIAM%2C+2008%2E)
41. José Grimm, Loı̄c Pottier, and Nicole Rostaing-Schmidt. Optimal time and minimum spacetime product for reversing a certain class of programs. PhD thesis, INRIA, 1996. [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Grimm%2C+Lo%C4%B1%CC%84c+Pottier%2C+and+Nicole+Rostaing%2DSchmidt%2E+Optimal+time+and+minimum+spacetime+product+for+reversing+a+certain+class+of+programs%2E+PhD+thesis%2C+INRIA%2C+1996%2E)
42. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, 2020. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+the+58)
43. Davood Hajinezhad and Michael M Zavlanos. Gradient-free multi-agent nonconvex nonsmooth optimization. In 2018 IEEE Conference on Decision and Control (CDC), pages 4939–4944, 2018. [find](https://scholar.google.com/scholar?q=Davood+Hajinezhad+and+Michael+M+Zavlanos%2E+Gradient%2Dfree+multi%2Dagent+nonconvex+nonsmooth+optimization%2E+In+2018+IEEE+Conference+on+Decision+and+Control+%28CDC%29%2C+pages+4939%E2%80%934944%2C+2018%2E)
44. Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022. — [arXiv:2212.13345](https://arxiv.org/abs/2212.13345) [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2E+The+forward%2Dforward+algorithm%3A+Some+preliminary+investigations%2E+arXiv+preprint+arXiv%3A2212%2E13345%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E13345%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2212%2E13345%29)
45. Bairu Hou, Joe O’Connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. Promptboosting: Black-box text classification with ten forward passes. arXiv preprint arXiv:2212.09257, 2022. — [arXiv:2212.09257](https://arxiv.org/abs/2212.09257) [find](https://scholar.google.com/scholar?q=Bairu+Hou%2C+Joe+O%E2%80%99Connor%2C+Jacob+Andreas%2C+Shiyu+Chang%2C+and+Yang+Zhang%2E+Promptboosting%3A+Black%2Dbox+text+classification+with+ten+forward+passes%2E+arXiv+preprint+arXiv%3A2212%2E09257%2C+2022%2E+%E2%80%94+%5BarXiv%3A2212%2E09257%5D%28)
46. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022.
47. Kevin G Jamieson, Robert Nowak, and Ben Recht. Query complexity of derivative-free optimization. In Advances in Neural Information Processing Systems, volume 25, 2012. [find](https://scholar.google.com/scholar?q=Kevin+G+Jamieson%2C+Robert+Nowak%2C+and+Ben+Recht%2E+Query+complexity+of+derivative%2Dfree+optimization%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+25%2C+2012%2E)
48. Kaiyi Ji, Zhe Wang, Yi Zhou, and Yingbin Liang. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. In International conference on machine learning, pages 3100–3109, 2019. [find](https://scholar.google.com/scholar?q=Kaiyi+Ji%2C+Zhe+Wang%2C+Yi+Zhou%2C+and+Yingbin+Liang%2E+Improved+zeroth%2Dorder+variance+reduced+algorithms+and+analysis+for+nonconvex+optimization%2E+In+International+conference+on+machine+learning%2C+pages+3100%E2%80%933)
49. Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26, 2013. [find](https://scholar.google.com/scholar?q=Rie+Johnson+and+Tong+Zhang%2E+Accelerating+stochastic+gradient+descent+using+predictive+variance+reduction%2E+In+C%2EJ%2E+Burges%2C+L%2E+Bottou%2C+M%2E+Welling%2C+Z%2E+Ghahramani%2C+and+K%2EQ%2E+Weinberger%2C+editors%2C+Advances+i)
50. Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition, 2020. [find](https://scholar.google.com/scholar?q=Hamed+Karimi%2C+Julie+Nutini%2C+and+Mark+Schmidt%2E+Linear+convergence+of+gradient+and+proximal%2Dgradient+methods+under+the+polyak%2D%C5%82ojasiewicz+condition%2C+2020%2E)
51. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human…
52. Diederik P Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In International Conference on Learning Representations, 2015.
53. Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Ananya+Kumar%2C+Aditi+Raghunathan%2C+Robbie+Matthew+Jones%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+Fine%2Dtuning+can+distort+pretrained+features+and+underperform+out%2Dof%2Ddistribution%2E+In+International+Conference+on+Lear)
54. Brian Lester, Rami Al-Rfou, and Noah Constant. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
55. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2C+2012%2E)
56. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. [Measuring the intrinsic dimension of objective landscapes](https://md.research.elamur.ai/free/en/papers/li-2018-measuring-intrinsic-dimension-objective-landscapes/). In International Conference on Learning Representations, 2018.
57. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597,… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Inte)
58. Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling SGD with stochastic differential equations (SDEs). In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Sadhika+Malladi%2C+and+Sanjeev+Arora%2E+On+the+validity+of+modeling+SGD+with+stochastic+differential+equations+%28SDEs%29%2E+In+A%2E+Beygelzimer%2C+Y%2E+Dauphin%2C+P%2E+Liang%2C+and+J%2E+Wortman+Vaughan%2C+editors%2C)
59. Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In International Conference on Machine Learning, pages 12656–12684, 2022. [find](https://scholar.google.com/scholar?q=Zhiyuan+Li%2C+Srinadh+Bhojanapalli%2C+Manzil+Zaheer%2C+Sashank+Reddi%2C+and+Sanjiv+Kumar%2E+Robust+training+of+neural+networks+using+scale+invariant+architectures%2E+In+International+Conference+on+Machine+Learnin)
60. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. [What makes good in-context examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, 2022…
61. Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. [Understanding the difficulty of training transformers](https://md.research.elamur.ai/free/en/papers/liu-2020-understanding-difficulty-training-transformers/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747–5763, 2020.
62. Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth-order stochastic variance reduction for nonconvex optimization. In Advances in Neural Information Processing Systems, volume 31, 2018. [find](https://scholar.google.com/scholar?q=Sijia+Liu%2C+Bhavya+Kailkhura%2C+Pin%2DYu+Chen%2C+Paishun+Ting%2C+Shiyu+Chang%2C+and+Lisa+Amini%2E+Zeroth%2Dorder+stochastic+variance+reduction+for+nonconvex+optimization%2E+In+Advances+in+Neural+Information+Processing)
63. Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signSGD via zeroth-order oracle. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Sijia+Liu%2C+Pin%2DYu+Chen%2C+Xiangyi+Chen%2C+and+Mingyi+Hong%2E+signSGD+via+zeroth%2Dorder+oracle%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
64. Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43–54, 2020. [find](https://scholar.google.com/scholar?q=Sijia+Liu%2C+Pin%2DYu+Chen%2C+Bhavya+Kailkhura%2C+Gaoyuan+Zhang%2C+Alfred+O+Hero+III%2C+and+Pramod+K+Varshney%2E+A+primer+on+zeroth%2Dorder+optimization+in+signal+processing+and+machine+learning%3A+Principals%2C+recent+a)
65. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
66. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8…
67. Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. arXiv preprint arXiv:2210.05643, 2022. [find](https://scholar.google.com/scholar?q=Sadhika+Malladi%2C+Alexander+Wettig%2C+Dingli+Yu%2C+Danqi+Chen%2C+and+Sanjeev+Arora%2E+A+kernel%2Dbased+view+of+language+model+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2210%2E05643%2C+2022%2E)
68. Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, 2018. [find](https://scholar.google.com/scholar?q=Horia+Mania%2C+Aurelia+Guy%2C+and+Benjamin+Recht%2E+Simple+random+search+of+static+linear+policies+is+competitive+for+reinforcement+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+31%2C)
69. Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983. [find](https://scholar.google.com/scholar?q=Arkadij+Semenovic%CC%8C+Nemirovskij+and+David+Borisovich+Yudin%2E+Problem+complexity+and+method+efficiency+in+optimization%2E+1983%2E)
70. Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17:527–566, 2017. [find](https://scholar.google.com/scholar?q=Yurii+Nesterov+and+Vladimir+Spokoiny%2E+Random+gradient%2Dfree+minimization+of+convex+functions%2E+Foundations+of+Computational+Mathematics%2C+17%3A527%E2%80%93566%2C+2017%2E)
71. Deniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson, and Ryan P Adams. Randomized automatic differentiation. arXiv preprint arXiv:2007.10412, 2020. — [arXiv:2007.10412](https://arxiv.org/abs/2007.10412) [find](https://scholar.google.com/scholar?q=Deniz+Oktay%2C+Nick+McGreivy%2C+Joshua+Aduol%2C+Alex+Beatson%2C+and+Ryan+P+Adams%2E+Randomized+automatic+differentiation%2E+arXiv+preprint+arXiv%3A2007%2E10412%2C+2020%2E+%E2%80%94+%5BarXiv%3A2007%2E10412%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2007%2E1)
72. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
73. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
74. Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv preprint arXiv:1811.07062, 2018. — [arXiv:1811.07062](https://arxiv.org/abs/1811.07062) [find](https://scholar.google.com/scholar?q=Vardan+Papyan%2E+The+full+spectrum+of+deepnet+hessians+at+scale%3A+Dynamics+with+sgd+training+and+sample+size%2E+arXiv+preprint+arXiv%3A1811%2E07062%2C+2018%2E+%E2%80%94+%5BarXiv%3A1811%2E07062%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1811%2E07062%29)
75. Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252):1–64, 2020. [find](https://scholar.google.com/scholar?q=Vardan+Papyan%2E+Traces+of+class%2Fcross%2Dclass+structure+pervade+deep+learning+spectra%2E+Journal+of+Machine+Learning+Research%2C+21%28252%29%3A1%E2%80%9364%2C+2020%2E)
76. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
77. Mohammad Taher Pilehvar and Jose Camacho-Collados. [WiC: the word-in-context dataset for evaluating context-sensitive meaning representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (L…
78. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. [Grips: Gradient-free, edit-based instruction search for prompting large language models](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/). arXiv preprint arXiv:2203.07281, 2022.
79. Maxim Raginsky and Alexander Rakhlin. Information-based complexity, feedback and dynamics in convex programming. IEEE Transactions on Information Theory, 57(10):7036– 7056, 2011. [find](https://scholar.google.com/scholar?q=Maxim+Raginsky+and+Alexander+Rakhlin%2E+Information%2Dbased+complexity%2C+feedback+and+dynamics+in+convex+programming%2E+IEEE+Transactions+on+Information+Theory%2C+57%2810%29%3A7036%E2%80%93+7056%2C+2011%2E)
80. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016.
81. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+2011%2E)
82. Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454, 2017. — [arXiv:1706.04454](https://arxiv.org/abs/1706.04454) [find](https://scholar.google.com/scholar?q=Levent+Sagun%2C+Utku+Evci%2C+V+Ugur+Guney%2C+Yann+Dauphin%2C+and+Leon+Bottou%2E+Empirical+analysis+of+the+hessian+of+over%2Dparametrized+neural+networks%2E+arXiv+preprint+arXiv%3A1706%2E04454%2C+2017%2E+%E2%80%94+%5BarXiv%3A1706%2E04454)
83. Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Nikunj+Saunshi%2C+Sadhika+Malladi%2C+and+Sanjeev+Arora%2E+A+mathematical+exploration+of+why+language+models+help+solve+downstream+tasks%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
84. Timo Schick and Hinrich Schütze. [Exploiting cloze-questions for few-shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, 2021.
85. Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. The Journal of Machine Learning Research, 18(1):1703–1713, 2017. [find](https://scholar.google.com/scholar?q=Ohad+Shamir%2E+An+optimal+algorithm+for+bandit+and+zero%2Dorder+convex+optimization+with+two%2Dpoint+feedback%2E+The+Journal+of+Machine+Learning+Research%2C+18%281%29%3A1703%E2%80%931713%2C+2017%2E)
86. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
87. James C Spall. A one-measurement form of simultaneous perturbation stochastic approximation. Automatica, 33(1):109–112, 1997. [find](https://scholar.google.com/scholar?q=James+C+Spall%2E+A+one%2Dmeasurement+form+of+simultaneous+perturbation+stochastic+approximation%2E+Automatica%2C+33%281%29%3A109%E2%80%93112%2C+1997%2E)
88. J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3):332–341, 1992. [find](https://scholar.google.com/scholar?q=J%2EC%2E+Spall%2E+Multivariate+stochastic+approximation+using+a+simultaneous+perturbation+gradient+approximation%2E+IEEE+Transactions+on+Automatic+Control%2C+37%283%29%3A332%E2%80%93341%2C+1992%2E)
89. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+In+Advances+in+Neural+In)
90. Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. BBTv2: Towards a gradient-free future with large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3916–3930, 2022. [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Zhengfu+He%2C+Hong+Qian%2C+Yunhua+Zhou%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+BBTv2%3A+Towards+a+gradient%2Dfree+future+with+large+language+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+)
91. Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. In International Conference on Machine Learning, pages 20841–20855, 2022. [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Yunfan+Shao%2C+Hong+Qian%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+Black%2Dbox+tuning+for+language%2Dmodel%2Das%2Da%2Dservice%2E+In+International+Conference+on+Machine+Learning%2C+pages+20841%E2%80%9320855%2C+2022%2E)
92. Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meProp: Sparsified back propagation for accelerated deep learning with reduced overfitting. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 3299–3308, 2017. [find](https://scholar.google.com/scholar?q=Xu+Sun%2C+Xuancheng+Ren%2C+Shuming+Ma%2C+and+Houfeng+Wang%2E+meProp%3A+Sparsified+back+propagation+for+accelerated+deep+learning+with+reduced+overfitting%2E+In+Proceedings+of+the+34th+International+Conference+on+)
93. Yujie Tang and Na Li. Distributed zero-order algorithms for nonconvex multi-agent optimization. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 781–786, 2019. [find](https://scholar.google.com/scholar?q=Yujie+Tang+and+Na+Li%2E+Distributed+zero%2Dorder+algorithms+for+nonconvex+multi%2Dagent+optimization%2E+In+2019+57th+Annual+Allerton+Conference+on+Communication%2C+Control%2C+and+Computing+%28Allerton%29%2C+pages+781%E2%80%937)
94. Zhiwei Tang, Dmitry Rybin, and Tsung-Hui Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles, 2023. [find](https://scholar.google.com/scholar?q=Zhiwei+Tang%2C+Dmitry+Rybin%2C+and+Tsung%2DHui+Chang%2E+Zeroth%2Dorder+optimization+meets+human+feedback%3A+Provable+learning+via+ranking+oracles%2C+2023%2E)
95. Alexander Timurovich Vakhitov, Oleg Nikolaevich Granichin, and LS Gurevich. Algorithm for stochastic approximation with trial input perturbation in the nonstationary problem of optimization. Automation and Remote Control, 70:1827–1835, 2009. [find](https://scholar.google.com/scholar?q=Alexander+Timurovich+Vakhitov%2C+Oleg+Nikolaevich+Granichin%2C+and+LS+Gurevich%2E+Algorithm+for+stochastic+approximation+with+trial+input+perturbation+in+the+nonstationary+problem+of+optimization%2E+Automatio)
96. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, volume 30, 2017.
97. Ellen M Voorhees and Dawn M Tice. Building a question answering test collection. In the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, 2000. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees+and+Dawn+M+Tice%2E+Building+a+question+answering+test+collection%2E+In+the+23rd+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+retrieval%2C+2000%2E)
98. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in neural information processing systems, volume 32, 2019.
99. Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+Xi+Chen%2C+Alexander+J+Smola%2C+and+Eric+P+Xing%2E+Variance+reduction+for+stochastic+gradient+optimization%2E+In+C%2EJ%2E+Burges%2C+L%2E+Bottou%2C+M%2E+Welling%2C+Z%2E+Ghahramani%2C+and+K%2EQ%2E+Weinberger%2C+editors%2C+Ad)
100. Yining Wang, Simon Du, Sivaraman Balakrishnan, and Aarti Singh. Stochastic zeroth-order optimization in high dimensions. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84, pages 1356–1365, 2018. [find](https://scholar.google.com/scholar?q=Yining+Wang%2C+Simon+Du%2C+Sivaraman+Balakrishnan%2C+and+Aarti+Singh%2E+Stochastic+zeroth%2Dorder+optimization+in+high+dimensions%2E+In+Proceedings+of+the+Twenty%2DFirst+International+Conference+on+Artificial+Intel)
101. Zhongruo Wang, Krishnakumar Balasubramanian, Shiqian Ma, and Meisam Razaviyayn. Zeroth-order algorithms for nonconvex minimax problems with improved complexities. arXiv preprint arXiv:2001.07819, 2020. — [arXiv:2001.07819](https://arxiv.org/abs/2001.07819) [find](https://scholar.google.com/scholar?q=Zhongruo+Wang%2C+Krishnakumar+Balasubramanian%2C+Shiqian+Ma%2C+and+Meisam+Razaviyayn%2E+Zeroth%2Dorder+algorithms+for+nonconvex+minimax+problems+with+improved+complexities%2E+arXiv+preprint+arXiv%3A2001%2E07819%2C+2020)
102. Bingzhen Wei, Xu Sun, Xuancheng Ren, and Jingjing Xu. Minimal effort back propagation for convolutional neural networks. arXiv preprint arXiv:1709.05804, 2017. — [arXiv:1709.05804](https://arxiv.org/abs/1709.05804) [find](https://scholar.google.com/scholar?q=Bingzhen+Wei%2C+Xu+Sun%2C+Xuancheng+Ren%2C+and+Jingjing+Xu%2E+Minimal+effort+back+propagation+for+convolutional+neural+networks%2E+arXiv+preprint+arXiv%3A1709%2E05804%2C+2017%2E+%E2%80%94+%5BarXiv%3A1709%2E05804%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
103. Adina Williams, Nikita Nangia, and Samuel Bowman. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),…
104. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
105. Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, and Rong Ge. Dissecting hessian: Understanding common structure of hessian in neural networks. arXiv preprint arXiv:2010.04261, 2020. — [arXiv:2010.04261](https://arxiv.org/abs/2010.04261) [find](https://scholar.google.com/scholar?q=Yikai+Wu%2C+Xingyu+Zhu%2C+Chenwei+Wu%2C+Annie+Wang%2C+and+Rong+Ge%2E+Dissecting+hessian%3A+Understanding+common+structure+of+hessian+in+neural+networks%2E+arXiv+preprint+arXiv%3A2010%2E04261%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E04261%5D)
106. Jiaxi Yang, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Iterative forward tuning boosts in-context learning in language models, 2023. [find](https://scholar.google.com/scholar?q=Jiaxi+Yang%2C+Binyuan+Hui%2C+Min+Yang%2C+Binhua+Li%2C+Fei+Huang%2C+and+Yongbin+Li%2E+Iterative+forward+tuning+boosts+in%2Dcontext+learning+in+language+models%2C+2023%2E)
107. Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581–590, 2020. [find](https://scholar.google.com/scholar?q=Zhewei+Yao%2C+Amir+Gholami%2C+Kurt+Keutzer%2C+and+Michael+W+Mahoney%2E+Pyhessian%3A+Neural+networks+through+the+lens+of+the+hessian%2E+In+2020+IEEE+international+conference+on+big+data+%28Big+data%29%2C+pages+581%E2%80%93590%2C+)
108. Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack. arXiv preprint arXiv:1812.11377, 2018. — [arXiv:1812.11377](https://arxiv.org/abs/1812.11377) [find](https://scholar.google.com/scholar?q=Haishan+Ye%2C+Zhichao+Huang%2C+Cong+Fang%2C+Chris+Junchi+Li%2C+and+Tong+Zhang%2E+Hessian%2Daware+zeroth%2Dorder+optimization+for+black%2Dbox+adversarial+attack%2E+arXiv+preprint+arXiv%3A1812%2E11377%2C+2018%2E+%E2%80%94+%5BarXiv%3A1812%2E11)
109. Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017. — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) [find](https://scholar.google.com/scholar?q=Yang+You%2C+Igor+Gitman%2C+and+Boris+Ginsburg%2E+Large+batch+training+of+convolutional+networks%2E+arXiv+preprint+arXiv%3A1708%2E03888%2C+2017%2E+%E2%80%94+%5BarXiv%3A1708%2E03888%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E03888%29)
110. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019. [find](https://scholar.google.com/scholar?q=Yang+You%2C+Jing+Li%2C+Sashank+Reddi%2C+Jonathan+Hseu%2C+Sanjiv+Kumar%2C+Srinadh+Bhojanapalli%2C+Xiaodan+Song%2C+James+Demmel%2C+Kurt+Keutzer%2C+and+Cho%2DJui+Hsieh%2E+Large+batch+optimization+for+deep+learning%3A+Training+b)
111. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. [Record: Bridging the gap between human and machine commonsense reading comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/). arXiv preprint arXiv:1810.12885, 2018.
112. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.
113. Yan Zhang, Yi Zhou, Kaiyi Ji, and Michael M Zavlanos. A new one-point residual-feedback oracle for black-box learning and control. Automatica, 136:110006, 2022. [find](https://scholar.google.com/scholar?q=Yan+Zhang%2C+Yi+Zhou%2C+Kaiyi+Ji%2C+and+Michael+M+Zavlanos%2E+A+new+one%2Dpoint+residual%2Dfeedback+oracle+for+black%2Dbox+learning+and+control%2E+Automatica%2C+136%3A110006%2C+2022%2E)

*Updated 2 November 2026*