---
title: "Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer"
description: "The authors propose Pen2Sword, in which a small domain expert model (the \"Pen\") passes its knowledge to a larger target… 34 references."
authors: ["Meishu Peng", "Ziyue Zhang", "Yi Zhang", "Pengyang Wang", "Zixuan Yuan", "Denghui Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.5/"
doi: "10.18653/v1/2026.findings-acl.5"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peng-2026-accelerating-llm-fine-tuning-embedding/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2026-accelerating-llm-fine-tuning-embedding/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer

# Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer

**Authors:** Meishu Peng, Ziyue Zhang, [Yi Zhang](https://md.research.elamur.ai/free/en/authors/yi-zhang/), Pengyang Wang, Zixuan Yuan and [Denghui Zhang](https://md.research.elamur.ai/free/en/authors/denghui-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.5/) · [PDF](https://aclanthology.org/2026.findings-acl.5.pdf) · [DOI 10.18653/v1/2026.findings-acl.5](https://doi.org/10.18653/v1/2026.findings-acl.5) · [OpenAlex](https://openalex.org/W7166840366)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Fine-tuning large language models for a new domain is costly. PEFT methods such as LoRA update only Transformer block weights and ignore the embedding layer. Weak-to-Strong approaches need pseudo-labels, which require running inference with a small model and keeping both models in GPU memory. The authors propose Pen2Sword, in which a small domain expert model (the "Pen") passes its knowledge to a larger target model (the "Sword") only through the embedding matrix. The framework has three stages:

1. A preserving function picks the expert whose embeddings best match the target domain.
2. The two vocabularies are merged into a union, and the expert's hidden dimension is expanded to match the target's.
3. The two embeddings are combined by weighted fusion, and the target is then fine-tuned starting from this fused embedding.

Across code, math and general-knowledge tasks, the method converges faster and reaches higher final scores than vanilla models, LoRA, Weak2Strong and ProxyTuning. It also works with several model families and PEFT variants. The work matters because it offers a cheap way to reuse existing small specialist models, with no pseudo-labels and no requirement for identical architectures.

## Key points
- Evaluated on HumanEval, MATH and MMLU, with reported gains of +13.6% in code and +20.1% in math. The method also generalizes to unseen domains such as Physics and Politics.
- Tested on Llama, Vicuna and Baichuan models. Vocabularies can differ within a family; for example, CodeLlama-7B has 16 more special tokens than Llama2-13B-Base.
- The ratio of expert to target hidden size matters:
  - About 80% (e.g., 7B→13B) gives the best trade-off.
  - 67% (1B→3B) clearly hurts performance.
  - 100% (7B→7B) adds no model capacity.
- With a fusion ratio of α = 0.5, Pen2Sword matches an 8k-step LoRA model after only 1k steps. Smaller α behaves much like LoRA, while larger α slows alignment.
- Limitations: the method has not been tested on models at 70B/405B scale or for inference, and it struggles when the dimension gap is large. Code is available at github.com/pengmeishu/Pen2Sword.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 33 works · 2,223 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 53 works · 1,367 citations
- [Stevens Institute of Technology](https://md.research.elamur.ai/free/en/universities/stevens-institute-of-technology/) — University, Hoboken, United States · 10 works · 95 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Meishu Peng, Ziyue Zhang, Yi Zhang, Pengyang Wang, Zixuan Yuan and Denghui Zhang. 2026. Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{peng2026accelerating,
  title = {{Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer}},
  author = {Meishu Peng, Ziyue Zhang, Yi Zhang, Pengyang Wang, Zixuan Yuan and Denghui Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.5},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.5}
}
```

- [← previous ACBQ: Adaptive Cross-Block Quantization of Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2026-acbq-adaptive-cross-block-quantization/)
- [next → Acceleration of Backpropagation in Linear Layers of Transformer Models Based on Gradient Structure](https://md.research.elamur.ai/free/en/papers/topchii-2026-acceleration-backpropagation-linear-layers-transformer/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/)

## References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 2016. Net2net: Accelerating learning via knowledge transfer. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Ian+J%2E+Goodfellow%2C+and+Jonathon+Shlens%2E+2016%2E+Net2net%3A+Accelerating+learning+via+knowledge+transfer%2E+In+4th+International+Conference+on+Learning+Representations%2C+ICLR+2016%2C+San+Juan%2C+Puer)
2. Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC bioinformatics, 20(1):511. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha+and+Dina+Demner%2DFushman%2E+2019%2E+A+question%2Dentailment+approach+to+question+answering%2E+BMC+bioinformatics%2C+20%281%29%3A511%2E)
3. Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. 2020. Low-rank bottleneck in multi-head attention models. In International conference on machine learning, pages 864–873. PMLR. [find](https://scholar.google.com/scholar?q=Srinadh+Bhojanapalli%2C+Chulhee+Yun%2C+Ankit+Singh+Rawat%2C+Sashank+Reddi%2C+and+Sanjiv+Kumar%2E+2020%2E+Low%2Drank+bottleneck+in+multi%2Dhead+attention+models%2E+In+International+conference+on+machine+learning%2C+pages+)
4. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
5. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). Advances in Neural Information Processing Systems, 36.
6. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
7. Konstantin Dobler and Gerard De Melo. 2023. [Focus: Effective embedding initialization for monolingual specialization of multilingual models](https://md.research.elamur.ai/free/en/papers/dobler-2023-focus-effective-embedding-initialization-monolingual/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440–13454.
8. Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023. [Scaling down to scale up: A guide to parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/lialin-2023-scaling-down-scale-up-guide/). arXiv preprint arXiv:2303.15647.
9. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2023%2E+Holistic+evaluation+of+language+models%2E+)
10. Yue Guo and Yi Yang. 2024. Improving weak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.19032. — [arXiv:2406.19032](https://arxiv.org/abs/2406.19032) [find](https://scholar.google.com/scholar?q=Yue+Guo+and+Yi+Yang%2E+2024%2E+Improving+weak%2Dto%2Dstrong+generalization+with+reliability%2Daware+alignment%2E+arXiv+preprint+arXiv%3A2406%2E19032%2E+%E2%80%94+%5BarXiv%3A2406%2E19032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2406%2E19032%29)
11. Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints. [find](https://scholar.google.com/scholar?q=Muhammad+Usman+Hadi%2C+Rizwan+Qureshi%2C+Abbas+Shah%2C+Muhammad+Irfan%2C+Anas+Zafar%2C+Muhammad+Bilal+Shaikh%2C+Naveed+Akhtar%2C+Jia+Wu%2C+Seyedali+Mirjalili%2C+et+al%2E+2023%2E+Large+language+models%3A+a+comprehensive+surve)
12. Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. 2024a. Tuning language models by proxy. CoRR, abs/2401.08565. [find](https://scholar.google.com/scholar?q=Alisa+Liu%2C+Xiaochuang+Han%2C+Yizhong+Wang%2C+Yulia+Tsvetkov%2C+Yejin+Choi%2C+and+Noah+A%2E+Smith%2E+2024a%2E+Tuning+language+models+by+proxy%2E+CoRR%2C+abs%2F2401%2E08565%2E)
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). volume abs/2009.03300.
14. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024b. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024b%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+In+Forty%2Dfirst+International+C)
15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). CoRR, abs/2103.03874.
16. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May… [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2024%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+I)
17. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
18. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. [Wechsel: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
19. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
20. Dhroov Pandey, Jonah Ghebremichael, Zongqing Qi, and Tong Shu. 2025. A comparative survey: Reusing small pre-trained models for efficient large model training. In Proceedings of the SC ’24 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’… [find](https://scholar.google.com/scholar?q=Dhroov+Pandey%2C+Jonah+Ghebremichael%2C+Zongqing+Qi%2C+and+Tong+Shu%2E+2025%2E+A+comparative+survey%3A+Reusing+small+pre%2Dtrained+models+for+efficient+large+model+training%2E+In+Proceedings+of+the+SC+%E2%80%9924+Workshops+o)
21. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). In The Eleventh International Conference on Learning Representations.
22. Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. ELLE: Efficient lifelong pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2789–2810, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Jiajie+Zhang%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2022%2E+ELLE%3A+Efficient+lifelong+pre%2Dtraining+for+emerging+data%2E+In+Findings+of+the+Association+for+Computational+Li)
23. Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang, and Kenji Kawaguchi. 2023. Self-distillation for further pre-training of transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Seanie+Lee%2C+Minki+Kang%2C+Juho+Lee%2C+Sung+Ju+Hwang%2C+and+Kenji+Kawaguchi%2E+2023%2E+Self%2Ddistillation+for+further+pre%2Dtraining+of+transformers%2E+In+The+Eleventh+International+Conference+on+Learning+Representat)
24. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Chong Li, Jiajun Zhang,… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
25. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. of the 202… [find](https://scholar.google.com/scholar?q=Le+Yu%2C+Mei+Li%2C+Mingfeng+Xue%2C+Pei+Zhang%2C+Qin+Zhu%2C+Rui+Men%2C+Runji+Lin%2C+Tianhao+Li%2C+Tianyi+Tang%2C+Tingyu+Xia%2C+Xingzhang+Ren%2C+Xuancheng+Ren%2C+Yang+Fan%2C+Yang+Su%2C+Yichang+Zhang%2C+Yu+Wan%2C+Yuqiong+Liu%2C+Zeyu+Cui%2C)
26. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, J… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
27. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Cur…
29. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inferenc…](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/)
30. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fang, Lei Su, Liang Song,… — [Baichuan 2: Open Large-scale Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
31. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vie…
32. Chiyuan Zhang, Samy Bengio, and Yoram Singer. 2022. Are all layers created equal? Journal of Machine Learning Research, 23(67):1–28. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+and+Yoram+Singer%2E+2022%2E+Are+all+layers+created+equal%3F+Journal+of+Machine+Learning+Research%2C+23%2867%29%3A1%E2%80%9328%2E)
33. Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogério Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. 2023. Learning to grow pretrained models for efficient transformer training. In The Eleventh International Conference on Learning Representations, I… [find](https://scholar.google.com/scholar?q=Peihao+Wang%2C+Rameswar+Panda%2C+Lucas+Torroba+Hennigen%2C+Philip+Greengard%2C+Leonid+Karlinsky%2C+Rog%C3%A9rio+Feris%2C+David+Daniel+Cox%2C+Zhangyang+Wang%2C+and+Yoon+Kim%2E+2023%2E+Learning+to+grow+pretrained+models+for+eff)
34. Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Yu Wu. 2024. [Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-let-expert-stick-his/). In Proceedings

*Updated 1 November 2026*