---
title: "Arcee's MergeKit: A Toolkit for Merging Large Language Models"
description: "The authors present MergeKit, Arcee's open-source library that implements community-developed… 53 references, cited by 98 papers here."
authors: ["Charles Goddard", "Shamane Siriwardhana", "Malikeh Ehghaghi", "Luke Meyers", "Vlad Karpukhin", "Brian Benedict", "Mark McQuade", "Jacob Solawetz"]
year: 2024
venue: "arXiv:2403.13257"
source: "https://arxiv.org/abs/2403.13257"
doi: "10.48550/arxiv.2403.13257"
arxiv: "2403.13257"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Arcee's MergeKit: A Toolkit for Merging Large Language Models

# Arcee's MergeKit: A Toolkit for Merging Large Language Models

**Authors:** Charles Goddard, [Shamane Siriwardhana](https://md.research.elamur.ai/free/en/authors/shamane-siriwardhana/), Malikeh Ehghaghi, Luke Meyers, [Vlad Karpukhin](https://md.research.elamur.ai/free/en/authors/vladimir-karpukhin/), Brian Benedict, Mark McQuade and Jacob Solawetz · **Year:** 2024 · **Venue:** arXiv:2403.13257 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.13257) · [PDF](https://arxiv.org/pdf/2403.13257) · [DOI 10.48550/arxiv.2403.13257](https://doi.org/10.48550/arxiv.2403.13257) · [arXiv 2403.13257](https://arxiv.org/abs/2403.13257) · [OpenAlex](https://openalex.org/W4393108834)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [98](https://md.research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Fine-tuning a separate LLM for every task is costly to store and deploy. It can also erode general abilities through catastrophic forgetting, and it stops related tasks from benefiting one another. Model merging addresses this by combining the parameters of existing checkpoints into a single model with no extra training. The authors present MergeKit, Arcee's open-source library that implements community-developed merging algorithms behind one YAML-driven interface. The paper also surveys the field and groups methods by whether the source models share an architecture and an initialization. MergeKit's design emphasizes interchangeable merge methods and tight integration with Hugging Face Transformers. It also uses out-of-core execution, loading only the tensors each step needs, so merges run on modest CPUs as well as GPU clusters. In a medical case study, Meditron-7B was merged with Llama2-7B chat. The LERP and SLERP variants beat both parent models on medical and general benchmarks, and SLERP came out slightly ahead. The toolkit matters because it made merging routine: thousands of community models, including some top open checkpoints, were built with it.

## Key points
- **Supported methods:** linear averaging (Model Soups), Task Arithmetic, TIES, DARE, Model Breadcrumbs and SLERP. Two further tools build new architectures:
  - the passthrough "FrankenMerging" method, used for Goliath-120b and as the first step of Depth Up-Scaling for SOLAR-10.7B;
  - `mergekit-moe`, which assembles "Franken-MoE" models from dense ones.
  - ZipIt, OT Fusion and Git Rebasin are planned.
- **Taxonomy of merging approaches:**
  - same architecture and initialization (LMC-based interpolation);
  - same architecture with different initializations (permutation or alignment methods);
  - different architectures (CALM, FUSELLM), which require additional training.
- **Medical case study:** SLERP scored 39.20 USMLE, 36.91 MedMCQA and 75.60 PubMedQA, versus 38.40, 24.07 and 71.40 for Meditron-7B. On MMLU, LERP reached 48.44, compared with 46.37 for Llama2-7B-Chat and 33.06 for Meditron.
- **Caveat on the results:** the DARE-TIES merge was weaker than the others (USMLE 36.37, which is below Meditron; MMLU 41.17). So the paper's claim that every merge beats Meditron on medical tests does not fully hold.
- **Extension points:** contributors work mainly with `merge_methods/base.py`, `graph.py`, `plan.py` and `architecture.py`. The code is at github.com/arcee-ai/mergekit.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade and Jacob Solawetz. 2024. Arcee's MergeKit: A Toolkit for Merging Large Language Models. arXiv:2403.13257.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{goddard2024arcee,
  title = {{Arcee's MergeKit: A Toolkit for Merging Large Language Models}},
  author = {Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade and Jacob Solawetz},
  year = {2024},
  doi = {10.48550/arxiv.2403.13257},
  eprint = {2403.13257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.13257}
}
```

- [← previous Towards Measuring and Modeling “Culture” in LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/adilazuarda-2024-measuring-modeling-culture-llms-survey/)
- [next → Who Wrote this Code? Watermarking for Code Generation](https://md.research.elamur.ai/free/en/papers/lee-2024-who-wrote-this-code-watermarking/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

53 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2022. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836. — [arXiv:2209.04836](https://arxiv.org/abs/2209.04836) [find](https://scholar.google.com/scholar?q=Samuel+K+Ainsworth%2C+Jonathan+Hayase%2C+and+Siddhartha+Srinivasa%2E+2022%2E+Git+re%2Dbasin%3A+Merging+models+modulo+permutation+symmetries%2E+arXiv+preprint+arXiv%3A2209%2E04836%2E+%E2%80%94+%5BarXiv%3A2209%2E04836%5D%28https%3A%2F%2Farxiv%2Eorg)
2. Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187. — [arXiv:2403.13187](https://arxiv.org/abs/2403.13187) [find](https://scholar.google.com/scholar?q=Takuya+Akiba%2C+Makoto+Shing%2C+Yujin+Tang%2C+Qi+Sun%2C+and+David+Ha%2E+2024%2E+Evolutionary+optimization+of+model+merging+recipes%2E+arXiv+preprint+arXiv%3A2403%2E13187%2E+%E2%80%94+%5BarXiv%3A2403%2E13187%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403)
3. Anton Alexandrov, Veselin Raychev, Mark Niklas Müller, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. Mitigating catastrophic forgetting in language transfer via model merging. arXiv preprint arXiv:2407.08699. [find](https://scholar.google.com/scholar?q=Anton+Alexandrov%2C+Veselin+Raychev%2C+Mark+Niklas+M%C3%BCller%2C+Ce+Zhang%2C+Martin+Vechev%2C+and+Kristina+Toutanova%2E+2024%2E+Mitigating+catastrophic+forgetting+in+language+transfer+via+model+merging%2E+arXiv+preprint+)
4. Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. 2024. Llm augmented llms: Expanding capabilities through composition. arXiv preprint arXiv:2401.02412. [find](https://scholar.google.com/scholar?q=Rachit+Bansal%2C+Bidisha+Samanta%2C+Siddharth+Dalmia%2C+Nitish+Gupta%2C+Shikhar+Vashishth%2C+Sriram+Ganapathy%2C+Abhishek+Bapna%2C+Prateek+Jain%2C+and+Partha+Talukdar%2E+2024%2E+Llm+augmented+llms%3A+Expanding+capabilities)
5. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. 2023. [Meditron-70b: Scaling medical pretraining for large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/). arXiv preprint arXiv:2311.16079.
6. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023. Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.09530. [find](https://scholar.google.com/scholar?q=Daixuan+Cheng%2C+Shaohan+Huang%2C+and+Furu+Wei%2E+2023%2E+Adapting+large+language+models+via+reading+comprehension%2E+arXiv+preprint+arXiv%3A2309%2E09530%2E)
7. Kyle Corbitt. 2023. How we built “mistral 7b fine-tune optimized,” the best 7b model for fine-tuning. [find](https://scholar.google.com/scholar?q=Kyle+Corbitt%2E+2023%2E+How+we+built+%E2%80%9Cmistral+7b+fine%2Dtune+optimized%2C%E2%80%9D+the+best+7b+model+for+fine%2Dtuning%2E)
8. MohammadReza Davari and Eugene Belilovsky. 2023. Model breadcrumbs: Scaling multi-task model merging with sparse masks. arXiv preprint arXiv:2312.06795. — [arXiv:2312.06795](https://arxiv.org/abs/2312.06795) [find](https://scholar.google.com/scholar?q=MohammadReza+Davari+and+Eugene+Belilovsky%2E+2023%2E+Model+breadcrumbs%3A+Scaling+multi%2Dtask+model+merging+with+sparse+masks%2E+arXiv+preprint+arXiv%3A2312%2E06795%2E+%E2%80%94+%5BarXiv%3A2312%2E06795%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2312)
9. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385. [find](https://scholar.google.com/scholar?q=Matthias+De+Lange%2C+Rahaf+Aljundi%2C+Marc+Masana%2C+Sarah+Parisot%2C+Xu+Jia%2C+Ale%C5%A1+Leonardis%2C+Gregory+Slabaugh%2C+and+Tinne+Tuytelaars%2E+2021%2E+A+continual+learning+survey%3A+Defying+forgetting+in+classification+ta)
10. Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2021. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296. — [arXiv:2110.06296](https://arxiv.org/abs/2110.06296) [find](https://scholar.google.com/scholar?q=Rahim+Entezari%2C+Hanie+Sedghi%2C+Olga+Saukh%2C+and+Behnam+Neyshabur%2E+2021%2E+The+role+of+permutation+invariance+in+linear+mode+connectivity+of+neural+networks%2E+arXiv+preprint+arXiv%3A2110%2E06296%2E+%E2%80%94+%5BarXiv%3A2110%2E)
11. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31. [find](https://scholar.google.com/scholar?q=Timur+Garipov%2C+Pavel+Izmailov%2C+Dmitrii+Podoprikhin%2C+Dmitry+P+Vetrov%2C+and+Andrew+G+Wilson%2E+2018%2E+Loss+surfaces%2C+mode+connectivity%2C+and+fast+ensembling+of+dnns%2E+Advances+in+neural+information+processing)
12. Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Jordi Bayarri-Planas, Adrian Tormos, Daniel Hinjos, Pablo Bernabeu-Perez, Anna Arias-Duart, Pablo Agustin Martin-Torres, Lucia Urcelay-Ganzabal, Marta Gonzalez-Mallo, et al. 2024. Aloe: A family of fine-tuned open healthcare llms. arXiv preprint arXiv:240… — [arXiv:2405.01886](https://arxiv.org/abs/2405.01886) [find](https://scholar.google.com/scholar?q=Ashwin+Kumar+Gururajan%2C+Enrique+Lopez%2DCuena%2C+Jordi+Bayarri%2DPlanas%2C+Adrian+Tormos%2C+Daniel+Hinjos%2C+Pablo+Bernabeu%2DPerez%2C+Anna+Arias%2DDuart%2C+Pablo+Agustin+Martin%2DTorres%2C+Lucia+Urcelay%2DGanzabal%2C+Marta+Gonz)
13. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). arXiv preprint arXiv:2212.04089.
14. Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. 2023. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719. — [arXiv:2310.05719](https://arxiv.org/abs/2310.05719) [find](https://scholar.google.com/scholar?q=Moritz+Imfeld%2C+Jacopo+Graldi%2C+Marco+Giordano%2C+Thomas+Hofmann%2C+Sotiris+Anagnostidis%2C+and+Sidak+Pal+Singh%2E+2023%2E+Transformer+fusion+with+optimal+transport%2E+arXiv+preprint+arXiv%3A2310%2E05719%2E+%E2%80%94+%5BarXiv%3A2310)
15. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. [Averaging weights leads to wider optima and better generalization](https://md.research.elamur.ai/free/en/papers/izmailov-2018-averaging-weights-leads-wider-optima/). arXiv preprint arXiv:1803.05407.
16. Vaishnavh Nagarajan and J Zico Kolter. 2019. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Vaishnavh+Nagarajan+and+J+Zico+Kolter%2E+2019%2E+Uniform+convergence+may+be+unable+to+explain+generalization+in+deep+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E)
17. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
18. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2021. What is being transferred in transfer learning? arXiv preprint arXiv:2008.11687. — [arXiv:2008.11687](https://arxiv.org/abs/2008.11687) [find](https://scholar.google.com/scholar?q=Behnam+Neyshabur%2C+Hanie+Sedghi%2C+and+Chiyuan+Zhang%2E+2021%2E+What+is+being+transferred+in+transfer+learning%3F+arXiv+preprint+arXiv%3A2008%2E11687%2E+%E2%80%94+%5BarXiv%3A2008%2E11687%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2008%2E11687%29)
19. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [PubMedQA: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
20. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849. [find](https://scholar.google.com/scholar?q=Xisen+Jin%2C+Xiang+Ren%2C+Daniel+Preotiuc%2DPietro%2C+and+Pengxiang+Cheng%2E+2022%2E+Dataless+knowledge+fusion+by+merging+weights+of+language+models%2E+arXiv+preprint+arXiv%3A2212%2E09849%2E)
21. Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. 2022. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08403. — [arXiv:2211.08403](https://arxiv.org/abs/2211.08403) [find](https://scholar.google.com/scholar?q=Keller+Jordan%2C+Hanie+Sedghi%2C+Olga+Saukh%2C+Rahim+Entezari%2C+and+Behnam+Neyshabur%2E+2022%2E+Repair%3A+Renormalizing+permuted+activations+for+interpolation+repair%2E+arXiv+preprint+arXiv%3A2211%2E08403%2E+%E2%80%94+%5BarXiv%3A2211)
22. Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. [Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling](https://md.research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/). arXiv preprint arXiv:2312.15166.
23. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv:2405.01535.
24. Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. Sparse upcycling: Training mixture-of-experts from dense checkpoints. [find](https://scholar.google.com/scholar?q=Aran+Komatsuzaki%2C+Joan+Puigcerver%2C+James+Lee%2DThorp%2C+Carlos+Riquelme+Ruiz%2C+Basil+Mustafa%2C+Joshua+Ainslie%2C+Yi+Tay%2C+Mostafa+Dehghani%2C+and+Neil+Houlsby%2E+2023%2E+Sparse+upcycling%3A+Training+mixture%2Dof%2Dexperts)
25. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. [Biomistral: A collection of open-source pretrained large language models for medical domains](https://md.research.elamur.ai/free/en/papers/labrak-2024-biomistral-collection-open-source-pretrained/). arXiv preprint arXiv:2402.10373.
26. Michael S Matena and Colin A Raffel. 2022. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). Advances in Neural Information Processing Systems, 35:17703– 17716.
27. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. [Large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2402.06196.
28. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Conference on health, inference, and learning, pages 248–260. PMLR.
29. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Advances in Neural Information Processing Systems, 36.
30. Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. 2023. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In International Conference on Machine Learning, pages 28656–28679. PMLR. [find](https://scholar.google.com/scholar?q=Alexandre+Ram%C3%A9%2C+Kartik+Ahuja%2C+Jianyu+Zhang%2C+Matthieu+Cord%2C+L%C3%A9on+Bottou%2C+and+David+Lopez%2DPaz%2E+2023%2E+Model+ratatouille%3A+Recycling+diverse+models+for+out%2Dof%2Ddistribution+generalization%2E+In+International+)
31. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
32. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). arXiv preprint arXiv:2110.08207.
33. Ken Shoemake. 1985. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245–254. [find](https://scholar.google.com/scholar?q=Ken+Shoemake%2E+1985%2E+Animating+rotation+with+quaternion+curves%2E+In+Proceedings+of+the+12th+annual+conference+on+Computer+graphics+and+interactive+techniques%2C+pages+245%E2%80%93254%2E)
34. Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045–22055. [find](https://scholar.google.com/scholar?q=Sidak+Pal+Singh+and+Martin+Jaggi%2E+2020%2E+Model+fusion+via+optimal+transport%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A22045%E2%80%9322055%2E)
35. Shamane Siriwardhana, Mark McQuade, Thomas Gauthier, Lucas Atkins, Fernando Fernandes Neto, Luke Meyers, Anneketh Vij, Tyler Odenthal, Charles Goddard, Mary MacCarthy, et al. 2024. Domain adaptation of llama3-70b-instruct through continual pre-training and model merging: A comprehensive evaluation.… — [arXiv:2406.14971](https://arxiv.org/abs/2406.14971) [find](https://scholar.google.com/scholar?q=Shamane+Siriwardhana%2C+Mark+McQuade%2C+Thomas+Gauthier%2C+Lucas+Atkins%2C+Fernando+Fernandes+Neto%2C+Luke+Meyers%2C+Anneketh+Vij%2C+Tyler+Odenthal%2C+Charles+Goddard%2C+Mary+MacCarthy%2C+et+al%2E+2024%2E+Domain+adaptation+o)
36. Joshua Smith and Michael Gashler. 2017. An investigation of how neural networks learn from the experiences of peers through periodic weight averaging. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 731–736. IEEE. [find](https://scholar.google.com/scholar?q=Joshua+Smith+and+Michael+Gashler%2E+2017%2E+An+investigation+of+how+neural+networks+learn+from+the+experiences+of+peers+through+periodic+weight+averaging%2E+In+2017+16th+IEEE+International+Conference+on+Mac)
37. George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. 2023. Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.03053. — [arXiv:2305.03053](https://arxiv.org/abs/2305.03053) [find](https://scholar.google.com/scholar?q=George+Stoica%2C+Daniel+Bolya%2C+Jakob+Bjorner%2C+Taylor+Hearn%2C+and+Judy+Hoffman%2E+2023%2E+Zipit%21+merging+models+from+different+tasks+without+training%2E+arXiv+preprint+arXiv%3A2305%2E03053%2E+%E2%80%94+%5BarXiv%3A2305%2E03053%5D%28htt)
38. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. [Pmc-llama: Further fine-tuning llama on medical papers](https://md.research.elamur.ai/free/en/papers/wu-2023-pmc-llama-towards-building-open/). arXiv preprint arXiv:2304.14454.
39. Mingxu Tao, Chen Zhang, Quzhe Huang, Tianyao Ma, Songfang Huang, Dongyan Zhao, and Yansong Feng. 2024. Unlocking the potential of model merging for low-resource languages. arXiv preprint arXiv:2407.03994. [find](https://scholar.google.com/scholar?q=Mingxu+Tao%2C+Chen+Zhang%2C+Quzhe+Huang%2C+Tianyao+Ma%2C+Songfang+Huang%2C+Dongyan+Zhao%2C+and+Yansong+Feng%2E+2024%2E+Unlocking+the+potential+of+model+merging+for+low%2Dresource+languages%2E+arXiv+preprint+arXiv%3A2407%2E03)
40. Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. 2024. Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415. [find](https://scholar.google.com/scholar?q=Chengyue+Wu%2C+Yukang+Gan%2C+Yixiao+Ge%2C+Zeyu+Lu%2C+Jiahao+Wang%2C+Ye+Feng%2C+Ping+Luo%2C+and+Ying+Shan%2E+2024%2E+Llama+pro%3A+Progressive+llama+with+block+expansion%2E+arXiv+preprint+arXiv%3A2401%2E02415%2E)
41. Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. 2020. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300–15311. [find](https://scholar.google.com/scholar?q=Norman+Tatro%2C+Pin%2DYu+Chen%2C+Payel+Das%2C+Igor+Melnyk%2C+Prasanna+Sattigeri%2C+and+Rongjie+Lai%2E+2020%2E+Optimizing+mode+connectivity+via+neuron+alignment%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A1)
42. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
43. Joachim Utans. 1996. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pages 133–138. Citeseer. [find](https://scholar.google.com/scholar?q=Joachim+Utans%2E+1996%2E+Weight+averaging+for+neural+networks+and+local+resampling+schemes%2E+In+Proc%2E+AAAI%2D96+Workshop+on+Integrating+Multiple+Learned+Models%2E+AAAI+Press%2C+pages+133%E2%80%93138%2E+Citeseer%2E)
44. Neha Verma and Maha Elbayad. 2024. Merging text transformer models from different initializations. arXiv preprint arXiv:2403.00986. — [arXiv:2403.00986](https://arxiv.org/abs/2403.00986) [find](https://scholar.google.com/scholar?q=Neha+Verma+and+Maha+Elbayad%2E+2024%2E+Merging+text+transformer+models+from+different+initializations%2E+arXiv+preprint+arXiv%3A2403%2E00986%2E+%E2%80%94+%5BarXiv%3A2403%2E00986%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E00986%29)
45. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491. [find](https://scholar.google.com/scholar?q=Fanqi+Wan%2C+Xinting+Huang%2C+Deng+Cai%2C+Xiaojun+Quan%2C+Wei+Bi%2C+and+Shuming+Shi%2E+2024%2E+Knowledge+fusion+of+large+language+models%2E+arXiv+preprint+arXiv%3A2401%2E10491%2E)
46. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. [Aligning large language models with human: A survey](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/). arXiv preprint arXiv:2307.12966.
47. Chengcheng Wei, Ze Chen, Songtan Fang, Jiarong He, and Max Gao. 2024. Opdai at semeval-2024 task 6: Small llms can accelerate hallucination detection with weakly supervised data. arXiv preprint arXiv:2402.12913. [find](https://scholar.google.com/scholar?q=Chengcheng+Wei%2C+Ze+Chen%2C+Songtan+Fang%2C+Jiarong+He%2C+and+Max+Gao%2E+2024%2E+Opdai+at+semeval%2D2024+task+6%3A+Small+llms+can+accelerate+hallucination+detection+with+weakly+supervised+data%2E+arXiv+preprint+arXiv%3A)
48. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Huggingface’s transformers: State-ofthe-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771.
49. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/). In In…
50. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. [Resolving interference when merging models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/). arXiv preprint arXiv:2306.01708.
51. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. [Ties-merging: Resolving interference when merging models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/). Advances in Neural Information Processing Systems, 36.
52. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. [Language models are super mario: Absorbing abilities from homologous models as a free lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/). arXiv preprint arXiv:2311.03099.
53. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385.

*Updated 24 October 2026*