---
title: "LoraHub — paper by Huang et al. (2023) — summary and ideas"
description: "The authors present LoraHub. 60 references, cited by 95 papers here. Among them: 7."
authors: ["Chengsong Huang", "Qian Liu", "Bill Yuchen Lin", "Tianyu Pang", "Chao Du", "Min Lin"]
year: 2023
venue: "arXiv:2307.13269"
source: "https://arxiv.org/abs/2307.13269"
doi: "10.48550/arxiv.2307.13269"
arxiv: "2307.13269"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

# LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

**Authors:** Chengsong Huang, [Qian Liu](https://md.research.elamur.ai/free/en/authors/qian-liu/), [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Tianyu Pang](https://md.research.elamur.ai/free/en/authors/tianyu-pang/), [Chao Du](https://md.research.elamur.ai/free/en/authors/chao-du/) and [Min Lin](https://md.research.elamur.ai/free/en/authors/min-lin/) · **Year:** 2023 · **Venue:** arXiv:2307.13269 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.13269) · [PDF](https://arxiv.org/pdf/2307.13269) · [DOI 10.48550/arxiv.2307.13269](https://doi.org/10.48550/arxiv.2307.13269) · [arXiv 2307.13269](https://arxiv.org/abs/2307.13269) · [OpenAlex](https://openalex.org/W4385292177)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [95](https://md.research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether LoRA adapters, each trained for one task, can be reused and combined so that a large language model handles a new task it has never seen. The authors present LoraHub. It first trains a separate LoRA module for each of many upstream tasks. For a new target task, it then needs only a few labelled examples (around five). LoraHub merges the modules element-wise into a single adapter, scaling each one by a coefficient that may be positive or negative. A gradient-free optimizer searches for coefficients that perform well on those few examples. The method adds no parameters, needs no backpropagation and relies on no manual choice of modules. Experiments use FLAN-T5 on the Big-Bench Hard (BBH) benchmark. LoraHub comes close to few-shot in-context learning in accuracy without placing demonstrations in the prompt, so each inference uses far fewer tokens. With favourable demonstration examples, its best-case results exceed those of in-context learning. The authors argue this could support a shared marketplace where people exchange adapters and users with modest hardware compose them for their own needs.

## Key points
- On BBH, the overview figure reports these average scores: zero-shot 27.0, few-shot in-context learning 37.5, and LoraHub 34.7. LoraHub's inference throughput is close to zero-shot.
- Composition sums the A matrices and the B matrices of all modules using the same weights w_i. This requires every module to share the same rank r. Merging too many modules makes the search space unstable, so the authors randomly select a subset of candidates.
- Coefficients are adapted per task, not per example, using only forward passes. A new BBH task can be handled in under a minute on one A100, and the procedure can even run on a CPU-only machine.
- Unlike task-vector merging, MoE or routing methods, LoraHub uses plain LoRA training and no multi-task training. Module selection is fully data-driven.
- The code is available at github.com/sail-sg/lorahub, and the authors have made their trained LoRA modules publicly available at huggingface.co/lorahub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations
- [Washington University in St. Louis](https://md.research.elamur.ai/free/en/universities/washington-university-in-st-louis/) — University, St Louis, United States · 5 works · 952 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du and Min Lin. 2023. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. arXiv:2307.13269.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{huang2023lorahub,
  title = {{LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition}},
  author = {Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du and Min Lin},
  year = {2023},
  doi = {10.48550/arxiv.2307.13269},
  eprint = {2307.13269},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13269}
}
```

- [← previous AnglE-optimized Text Embeddings](https://md.research.elamur.ai/free/en/papers/li-2023-angle-optimized-text-embeddings/)
- [next → Parallel Context Windows for Large Language Models](https://md.research.elamur.ai/free/en/papers/ratner-2023-parallel-context-windows-large-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

60 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Samuel+Ainsworth%2C+Jonathan+Hayase%2C+and+Siddhartha+Srinivasa%2E+Git+re%2Dbasin%3A+Merging+models+modulo+permutation+symmetries%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2023%2E)
2. Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. Input-tuning: Adapting unfamiliar inputs to frozen pre-trained models. ArXiv preprint, 2022. [find](https://scholar.google.com/scholar?q=Shengnan+An%2C+Yifei+Li%2C+Zeqi+Lin%2C+Qian+Liu%2C+Bei+Chen%2C+Qiang+Fu%2C+Weizhu+Chen%2C+Nanning+Zheng%2C+and+Jian%2DGuang+Lou%2E+Input%2Dtuning%3A+Adapting+unfamiliar+inputs+to+frozen+pre%2Dtrained+models%2E+ArXiv+preprint%2C+20)
3. Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In Proc. of ICLR, 2022. [find](https://scholar.google.com/scholar?q=Vamsi+Aribandi%2C+Yi+Tay%2C+Tal+Schuster%2C+Jinfeng+Rao%2C+Huaixiu+Steven+Zheng%2C+Sanket+Vaibhav+Mehta%2C+Honglei+Zhuang%2C+Vinh+Q%2E+Tran%2C+Dara+Bahri%2C+Jianmo+Ni%2C+Jai+Prakash+Gupta%2C+Kai+Hui%2C+Sebastian+Ruder%2C+and+Don)
4. Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya… — [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. [Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/). CoRR, abs/2305.14788, 2023. doi: 10.48550/ARXIV.2305. 14788. URL https://doi.org/10.48550/arXiv.2305.14788. [↗](https://doi.org/10.48550/arXiv.2305.14788)
7. Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. Adapter-Soup: Weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023, 2023. [find](https://scholar.google.com/scholar?q=Alexandra+Chronopoulou%2C+Matthew+Peters%2C+Alexander+Fraser%2C+and+Jesse+Dodge%2E+Adapter%2DSoup%3A+Weight+averaging+to+improve+generalization+of+pretrained+language+models%2E+In+Findings+of+the+Association+for+Co)
8. Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Ya… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+S%2E+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xinyun+C)
9. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke,… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
10. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. CoRR, abs/2307.06945, 2023. doi: 10. 48550/ARXIV.2307.06945. URL https://doi.org/10.48550/arXiv.2307.06945. [↗](https://doi.org/10.48550/arXiv.2307.06945)
11. Aryo Pradipta Gema, Luke Daines, Pasquale Minervini, and Beatrice Alex. Parameter-efficient fine-tuning of llama for the clinical domain. ArXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Aryo+Pradipta+Gema%2C+Luke+Daines%2C+Pasquale+Minervini%2C+and+Beatrice+Alex%2E+Parameter%2Defficient+fine%2Dtuning+of+llama+for+the+clinical+domain%2E+ArXiv+preprint%2C+2023%2E)
12. Nikolaus Hansen and Andreas Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. Proceedings of IEEE International Conference on Evolutionary Computation, 1996. [find](https://scholar.google.com/scholar?q=Nikolaus+Hansen+and+Andreas+Ostermeier%2E+Adapting+arbitrary+normal+mutation+distributions+in+evolution+strategies%3A+the+covariance+matrix+adaptation%2E+Proceedings+of+IEEE+International+Conference+on+Evol)
13. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In Proc. of ICLR, 2022.
14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proc. of ICLR, 2022.
15. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). In The Eleventh International Conference on Learning Representations, 2023.
16. Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 1991. [find](https://scholar.google.com/scholar?q=Robert+A%2E+Jacobs%2C+Michael+I%2E+Jordan%2C+Steven+J%2E+Nowlan%2C+and+Geoffrey+E%2E+Hinton%2E+Adaptive+mixtures+of+local+experts%2E+Neural+Computation%2C+1991%2E)
17. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In International Conference on Machine Learning, 2023. URL [https://api.semanticscholar.org/CorpusI…](https://api.semanticscholar.org/CorpusI…) [↗](https://api.semanticscholar.org/CorpusI…)
18. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [Llmlingua: Compressing prompts for accelerated inference of large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, December 2023a…
19. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/). CoRR, abs/2310.06839, 2023b. doi: 10.48550/ARXIV.2310.06839. URL https://doi.org/10.48550/arXiv.2310.06839. [↗](https://doi.org/10.48550/arXiv.2310.06839)
20. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Xisen+Jin%2C+Xiang+Ren%2C+Daniel+Preotiuc%2DPietro%2C+and+Pengxiang+Cheng%2E+Dataless+knowledge+fusion+by+merging+weights+of+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representations)
21. Hiroaki Kingetsu, Kenichi Kobayashi, and Taiji Suzuki. Neural network module decomposition and recomposition. ArXiv preprint, 2021. [find](https://scholar.google.com/scholar?q=Hiroaki+Kingetsu%2C+Kenichi+Kobayashi%2C+and+Taiji+Suzuki%2E+Neural+network+module+decomposition+and+recomposition%2E+ArXiv+preprint%2C+2021%2E)
22. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proc. of EMNLP, 2021. [find](https://scholar.google.com/scholar?q=Brian+Lester%2C+Rami+Al%2DRfou%2C+and+Noah+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proc%2E+of+EMNLP%2C+2021%2E)
23. Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. [Compressing context to enhance inference efficiency of large language models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, December 2023. URL [https://arxiv.or…](https://arxiv.or…) [↗](https://arxiv.or…)
24. Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. Unsupervised cross-task generalization via retrieval augmentation. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Kangmin+Tan%2C+Chris+Miller%2C+Beiwen+Tian%2C+and+Xiang+Ren%2E+Unsupervised+cross%2Dtask+generalization+via+retrieval+augmentation%2E+In+NeurIPS%2C+2022%2E)
25. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). ArXiv, abs/2205.05638, 2022. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:248693283. [↗](https://api.semanticscholar.org/)
26. Jialin Liu, A. Moreau, Mike Preuss, Baptiste Rozière, Jérémy Rapin, Fabien Teytaud, and Olivier Teytaud. Versatile black-box optimization. Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020. [find](https://scholar.google.com/scholar?q=Jialin+Liu%2C+A%2E+Moreau%2C+Mike+Preuss%2C+Baptiste+Rozie%CC%80re%2C+Je%CC%81re%CC%81my+Rapin%2C+Fabien+Teytaud%2C+and+Olivier+Teytaud%2E+Versatile+black%2Dbox+optimization%2E+Proceedings+of+the+2020+Genetic+and+Evolutionary+Computati)
27. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/), 2023.
28. Xingtai Lv, Ning Ding, Yujia Qin, Zhiyuan Liu, and Maosong Sun. Parameter-efficient weight ensembling facilitates task-level knowledge transfer. In Annual Meeting of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Xingtai+Lv%2C+Ning+Ding%2C+Yujia+Qin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Parameter%2Defficient+weight+ensembling+facilitates+task%2Dlevel+knowledge+transfer%2E+In+Annual+Meeting+of+the+Association+for+Computational+)
29. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/](https://github.com/) huggingface/peft, 2022. [↗](https://github.com/)
30. Michael Matena and Colin Raffel. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). ArXiv preprint, 2021.
31. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+MetaICL%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Comp)
32. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proc. of ACL, 2022. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+Proc%2E+of+ACL%2C+2022%2E)
33. Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. ArXiv preprint, 2023. [find](https://scholar.google.com/scholar?q=Mohammed+Muqeeth%2C+Haokun+Liu%2C+and+Colin+Raffel%2E+Soft+merging+of+experts+with+adaptive+routing%2E+ArXiv+preprint%2C+2023%2E)
34. OpenAI. ChatGPT. 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
35. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
36. Panupong Pasupat and Percy Liang. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In Proc. of ACL, 2015.
37. Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter-efficient modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Alessandro+Sordoni%2C+Yoshua+Bengio%2C+and+Siva+Reddy%2E+Combining+parameter%2Defficient+modules+for+task%2Dlevel+generalisation%2E+In+Proceedings+of+the+17th+Conference+of+the+European+Chapt)
38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 2020.
39. J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform. [https://GitHub.com/FacebookResearch/Nevergrad](https://GitHub.com/FacebookResearch/Nevergrad), 2018. [↗](https://GitHub.com/FacebookResearch/Nevergrad)
40. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonatha… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
41. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proc. of ICLR, 2017. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+V%2E+Le%2C+Geoffrey+E%2E+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+In+Proc%2E)
42. Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixture-of-experts meets instruction tuning:a… — [Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models](https://md.research.elamur.ai/free/en/papers/shen-2023-mixture-experts-meets-instruction-tuning/)
43. George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv, 2023. [find](https://scholar.google.com/scholar?q=George+Stoica%2C+Daniel+Bolya%2C+Jakob+Bjorner%2C+Taylor+Hearn%2C+and+Judy+Hoffman%2E+Zipit%21+merging+models+from+different+tasks+without+training%2E+arXiv%2C+2023%2E)
44. Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, B… [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Yunfan+Shao%2C+Hong+Qian%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+Black%2Dbox+tuning+for+language%2Dmodel%2Das%2Da%2Dservice%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+Szepesva%CC%81ri%2C+Gang+Niu%2C+)
45. Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, and Xuanjing Huang. Multitask pre-training of modular prompt for Chinese few-shot learning. In Proc. of ACL, 2023. [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Zhengfu+He%2C+Qin+Zhu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+Multitask+pre%2Dtraining+of+modular+prompt+for+Chinese+few%2Dshot+learning%2E+In+Proc%2E+of+ACL%2C+2023%2E)
46. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). ArXiv prepri…
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neur…
48. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. [AdaMix: Mixture-of-adaptations for parameter-efficient model tuning](https://md.research.elamur.ai/free/en/papers/wang-2022-adamix-mixture-adaptations-parameter-efficient/). In Proc. of EMNLP, 2022.
49. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In Proc. of ICLR, 2022. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y%2E+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M%2E+Dai%2C+and+Quoc+V%2E+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+Proc%2E+of+ICLR%2C+2022%2E)
50. Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. π-tuning: Transferring multimodal foundation models with optimal multi-task interpolation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Internationa… [find](https://scholar.google.com/scholar?q=Chengyue+Wu%2C+Teng+Wang%2C+Yixiao+Ge%2C+Zeyu+Lu%2C+Ruisong+Zhou%2C+Ying+Shan%2C+and+Ping+Luo%2E+%CF%80%2Dtuning%3A+Transferring+multimodal+foundation+models+with+optimal+multi%2Dtask+interpolation%2E+In+Andreas+Krause%2C+Emma+Br)
51. PMLR, 2023a. URL [https://proceedings.mlr.press/v202/wu23t.html](https://proceedings.mlr.press/v202/wu23t.html). [↗](https://proceedings.mlr.press/v202/wu23t.html)
52. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. [Bloomberggpt: A large language model for finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/). CoRR, abs/2303.17564, 2023b. doi: 10.48550/arXiv.2303. 17564. URL https://doi.org/10.48550/arXiv.2303.1756… [↗](https://doi.org/10.48550/arXiv.2303.1756…)
53. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. [TIES-merging: Resolving interference when merging models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) xtaX3WyCj1. [↗](https://openreview.net/forum?id=)
54. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proc. of EMNLP, 2021. [find](https://scholar.google.com/scholar?q=Qinyuan+Ye%2C+Bill+Yuchen+Lin%2C+and+Xiang+Ren%2E+CrossFit%3A+A+few%2Dshot+learning+challenge+for+cross%2Dtask+generalization+in+NLP%2E+In+Proc%2E+of+EMNLP%2C+2021%2E)
55. Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In Proc. of ICLR, 2019. [find](https://scholar.google.com/scholar?q=Chris+Zhang%2C+Mengye+Ren%2C+and+Raquel+Urtasun%2E+Graph+hypernetworks+for+neural+architecture+search%2E+In+Proc%2E+of+ICLR%2C+2019%2E)
56. Fan Zhang, Duyu Tang, Yong Dai, Cong Zhou, Shuangzhi Wu, and Shuming Shi. Skillnet-nlu: A sparsely activated model for general-purpose natural language understanding, 2022. [find](https://scholar.google.com/scholar?q=Fan+Zhang%2C+Duyu+Tang%2C+Yong+Dai%2C+Cong+Zhou%2C+Shuangzhi+Wu%2C+and+Shuming+Shi%2E+Skillnet%2Dnlu%3A+A+sparsely+activated+model+for+general%2Dpurpose+natural+language+understanding%2C+2022%2E)
57. Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. ArXiv preprint, 2023a. [find](https://scholar.google.com/scholar?q=Jinghan+Zhang%2C+Shiqi+Chen%2C+Junteng+Liu%2C+and+Junxian+He%2E+Composing+parameter%2Defficient+modules+with+arithmetic+operations%2E+ArXiv+preprint%2C+2023a%2E)
58. Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. ArXiv, abs/2308.03303, 2023b. URL [https://api.semanticscholar.org/CorpusID:260683267](https://api.semanticscholar.org/CorpusID:260683267). [↗](https://api.semanticscholar.org/CorpusID:260683267)
59. Wangchunshu Zhou, Yuchen Eleanor Jiang, Ryan Cotterell, and Mrinmaya Sachan. Efficient prompting via dynamic in-context learning. CoRR, abs/2305.11170, 2023. doi: 10.48550/ARXIV.2305.11170. URL https://doi.org/10.48550/arXiv.2305.11170. *Table 3: The top five beneficial LoRA modules for BBH tasks an… — [doi:10.48550/arxiv.2305.11170](https://doi.org/10.48550/arxiv.2305.11170) [↗](https://doi.org/10.48550/arXiv.2305.11170)
60. WIQA: Last Process RACE: Is this the Right Answer WIQA: First Process 0.72 0.68 0.63 28.1 30.8 28.1 AdversarialQA: BiDAF 0.61 25.1 WebQuestions: What is the Answer 0.58 27.0 Identifying the last step of a given process. Determining if given answer is correct. Identifying the first step of a given pr… [find](https://scholar.google.com/scholar?q=WIQA%3A+Last+Process+RACE%3A+Is+this+the+Right+Answer+WIQA%3A+First+Process+0%2E72+0%2E68+0%2E63+28%2E1+30%2E8+28%2E1+AdversarialQA%3A+BiDAF+0%2E61+25%2E1+WebQuestions%3A+What+is+the+Answer+0%2E58+27%2E0+Identifying+the+last+step+)

*Updated 24 October 2026*