---
title: "DivMerge — paper by Touayouch et al. (2026)"
description: "The authors (Orange Research and academic partners) propose DivMerge, which uses each fine-tuned model's output… 71 references."
authors: ["Brahim Touayouch", "Loïc Fosse", "Géraldine Damnati", "Gwénolé Lecorvé"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.337/"
doi: "10.18653/v1/2026.eacl-long.337"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/touayouch-2026-divmerge-divergence-based-model-merging/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/touayouch-2026-divmerge-divergence-based-model-merging/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/) / DivMerge: A divergence-based model merging method for multi-tasking

# DivMerge: A divergence-based model merging method for multi-tasking

**Authors:** Brahim Touayouch, Loïc Fosse, [Géraldine Damnati](https://md.research.elamur.ai/free/en/authors/geraldine-damnati/) and Gwénolé Lecorvé · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.337/) · [PDF](https://aclanthology.org/2026.eacl-long.337.pdf) · [DOI 10.18653/v1/2026.eacl-long.337](https://doi.org/10.18653/v1/2026.eacl-long.337) · [OpenAlex](https://openalex.org/W7140090950)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Merging several fine-tuned models into one multi-task model costs far less than joint multi-task training. However, tasks tend to interfere with each other, and this gets worse as more models are combined. A central obstacle is that merging has no explicit target distribution to optimize toward.

The authors (Orange Research and academic partners) propose DivMerge, which uses each fine-tuned model's output distribution as a substitute target. The method works with task vectors, meaning the parameter shifts from a shared base model, inside the Task Arithmetic framework. It iteratively learns merging coefficients, either one per task or one per layer. The learning objective is to reduce the Jensen-Shannon divergence between the merged model and every fine-tuned model, measured on unlabelled inputs from each task, so no reference labels are needed. The paper also links the approach theoretically to weight disentanglement and to conventional dataset-mixing multi-task learning.

When tested with autoregressive models on both classification and generation tasks, DivMerge outperforms earlier merging methods and loses less performance as the number of tasks grows. This makes it a label-free way to reuse specialized models shared publicly, for example on Hugging Face.

## Key points
- Baselines include Model Averaging, Multi-SLERP, TIES and AdaMerging, an entropy-based optimization method. DivMerge shows narrower confidence intervals than AdaMerging, indicating more stable results.
- Results are reported with the ANP metric on classification pairs such as CoLA+SST-2, QNLI+MNLI and RTE+MRPC, and on mixed classification/generation sets with increasing task counts (95% CIs).
- DivMerge beats the non-optimization baselines with only 25 unlabelled samples per task. This is about 0.4% of the fine-tuning training data and 5% of the validation set. Optimization converges smoothly without signs of overfitting.
- A gap to joint training remains: standard multi-task learning reaches an ANP of 0.98 in the classification setup.
- Stated limitations: only full fine-tuning was tested (no LoRA), and the method assumes access to samples from each task's true input distribution.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) — Research institute, Paris, France · 168 works · 62,369 citations
- [École Polytechnique](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique/) — University, Palaiseau, France · 14 works · 898 citations
- [Aix-Marseille Université](https://md.research.elamur.ai/free/en/universities/aix-marseille-universite/) — University, Marseille, France · 24 works · 39,160 citations
- [Institut Polytechnique de Paris](https://md.research.elamur.ai/free/en/universities/institut-polytechnique-de-paris/) — University, Palaiseau, France · 15 works · 1,612 citations
- [Laboratoire d’Informatique et Systèmes](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-et-systemes/) — Research institute, Marseille, France · 8 works · 106 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Brahim Touayouch, Loïc Fosse, Géraldine Damnati and Gwénolé Lecorvé. 2026. DivMerge: A divergence-based model merging method for multi-tasking. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{touayouch2026divmerge,
  title = {{DivMerge: A divergence-based model merging method for multi-tasking}},
  author = {Brahim Touayouch, Loïc Fosse, Géraldine Damnati and Gwénolé Lecorvé},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.337},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.337}
}
```

- [← previous Diversity and Homogenisation in Generative AI Translation: A Comparative Study of English-Dutch Translation Across Domains](https://md.research.elamur.ai/free/en/papers/shterionov-2026-diversity-homogenisation-generative-ai-translation/)
- [next → DM Doppelgangers: Implicit Connectives as eRST Signals](https://md.research.elamur.ai/free/en/papers/ai-2026-dm-doppelgangers-implicit-connectives-as/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/)

## References

71 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shun-ichi Amari. 2006. Differential geometry of statistical inference. In Proc. USSR-Japan Symposium. [find](https://scholar.google.com/scholar?q=Shun%2Dichi+Amari%2E+2006%2E+Differential+geometry+of+statistical+inference%2E+In+Proc%2E+USSR%2DJapan+Symposium%2E)
2. Shun-Ichi Amari. 2009. a-divergence is unique, belonging to both f-divergence and bregman divergence classes. IEEE Transactions on Information Theory, 55(11):4925–4931. [find](https://scholar.google.com/scholar?q=Shun%2DIchi+Amari%2E+2009%2E+a%2Ddivergence+is+unique%2C+belonging+to+both+f%2Ddivergence+and+bregman+divergence+classes%2E+IEEE+Transactions+on+Information+Theory%2C+55%2811%29%3A4925%E2%80%934931%2E)
3. Shun-ichi Amari and Hiroshi Nagaoka. 2000. Methods of information geometry. American Mathematical Soc. [find](https://scholar.google.com/scholar?q=Shun%2Dichi+Amari+and+Hiroshi+Nagaoka%2E+2000%2E+Methods+of+information+geometry%2E+American+Mathematical+Soc%2E)
4. Dana H Ballard. 1987. Modular learning in neural networks. In Proc. of the National conference on Artificial intelligence. [find](https://scholar.google.com/scholar?q=Dana+H+Ballard%2E+1987%2E+Modular+learning+in+neural+networks%2E+In+Proc%2E+of+the+National+conference+on+Artificial+intelligence%2E)
5. Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. 2005. Clustering with bregman divergences. Journal of machine learning research, 6(Oct):1705–1749. [find](https://scholar.google.com/scholar?q=Arindam+Banerjee%2C+Srujana+Merugu%2C+Inderjit+S+Dhillon%2C+and+Joydeep+Ghosh%2E+2005%2E+Clustering+with+bregman+divergences%2E+Journal+of+machine+learning+research%2C+6%28Oct%29%3A1705%E2%80%931749%2E)
6. Jonathan Baxter. 2000. A model of inductive bias learning. Journal of artificial intelligence research, 12:149–198. [find](https://scholar.google.com/scholar?q=Jonathan+Baxter%2E+2000%2E+A+model+of+inductive+bias+learning%2E+Journal+of+artificial+intelligence+research%2C+12%3A149%E2%80%93198%2E)
7. Shai Ben-David and Reba Schuller. 2003. Exploiting task relatedness for multiple task learning. In Proc. Annual Conference on Learning Theory. [find](https://scholar.google.com/scholar?q=Shai+Ben%2DDavid+and+Reba+Schuller%2E+2003%2E+Exploiting+task+relatedness+for+multiple+task+learning%2E+In+Proc%2E+Annual+Conference+on+Learning+Theory%2E)
8. Yaiza Bermudez, Gaetan Bisson, Iñaki Esnaola, and Samir Perlaza. 2025. Démonstrations des théorèmes folkloriques sur la dérivée de radon-nikodym. In GRETSI’25-XXXe Colloque Francophone de Traitement du Signal et des Images. [find](https://scholar.google.com/scholar?q=Yaiza+Bermudez%2C+Gaetan+Bisson%2C+I%C3%B1aki+Esnaola%2C+and+Samir+Perlaza%2E+2025%2E+D%C3%A9monstrations+des+th%C3%A9or%C3%A8mes+folkloriques+sur+la+d%C3%A9riv%C3%A9e+de+radon%2Dnikodym%2E+In+GRETSI%E2%80%9925%2DXXXe+Colloque+Francophone+de+Traitement+d)
9. Yochai Blau and Tomer Michaeli. 2018. The perception-distortion tradeoff. In Proc. of the IEEE conference on computer vision and pattern recognition. [find](https://scholar.google.com/scholar?q=Yochai+Blau+and+Tomer+Michaeli%2E+2018%2E+The+perception%2Ddistortion+tradeoff%2E+In+Proc%2E+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2E)
10. Yochai Blau and Tomer Michaeli. 2019. Rethinking lossy compression: The rate-distortion-perception tradeoff. In Proc. ICML. [find](https://scholar.google.com/scholar?q=Yochai+Blau+and+Tomer+Michaeli%2E+2019%2E+Rethinking+lossy+compression%3A+The+rate%2Ddistortion%2Dperception+tradeoff%2E+In+Proc%2E+ICML%2E)
11. Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. 2020. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In Proc. ECCV. [find](https://scholar.google.com/scholar?q=Malik+Boudiaf%2C+J%C3%A9r%C3%B4me+Rony%2C+Imtiaz+Masud+Ziko%2C+Eric+Granger%2C+Marco+Pedersoli%2C+Pablo+Piantanida%2C+and+Ismail+Ben+Ayed%2E+2020%2E+A+unifying+mutual+information+view+of+metric+learning%3A+cross%2Dentropy+vs%2E+pair)
12. Lev M Bregman. 1967. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200–217. [find](https://scholar.google.com/scholar?q=Lev+M+Bregman%2E+1967%2E+The+relaxation+method+of+finding+the+common+point+of+convex+sets+and+its+application+to+the+solution+of+problems+in+convex+programming%2E+USSR+computational+mathematics+and+mathemat)
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Proc. NEURIPS.
14. Rich Caruana. 1997. Multitask Learning. Machine learning, 28(1):41–75. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1997%2E+Multitask+Learning%2E+Machine+learning%2C+28%281%29%3A41%E2%80%9375%2E)
15. Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. 2024. Language and task arithmetic with parameter-efficient layers for zero-shot summarization. In Proc. of the Fourth Workshop on Multilingual Representation Learning. [find](https://scholar.google.com/scholar?q=Alexandra+Chronopoulou%2C+Jonas+Pfeiffer%2C+Joshua+Maynez%2C+Xinyi+Wang%2C+Sebastian+Ruder%2C+and+Priyanka+Agrawal%2E+2024%2E+Language+and+task+arithmetic+with+parameter%2Defficient+layers+for+zero%2Dshot+summarization)
16. T. M. Cover and J. A. Thomas. 1991. Elements of Information Theory. John Wiley & Sons, Inc. [find](https://scholar.google.com/scholar?q=T%2E+M%2E+Cover+and+J%2E+A%2E+Thomas%2E+1991%2E+Elements+of+Information+Theory%2E+John+Wiley+%26+Sons%2C+Inc%2E)
17. H. Crauel. 2002. Random Probability Measures on Polish Spaces. Taylor & Francis. [find](https://scholar.google.com/scholar?q=H%2E+Crauel%2E+2002%2E+Random+Probability+Measures+on+Polish+Spaces%2E+Taylor+%26+Francis%2E)
18. Imre Csiszár. 1975. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pages 146–158. [find](https://scholar.google.com/scholar?q=Imre+Csisz%C3%A1r%2E+1975%2E+I%2Ddivergence+geometry+of+probability+distributions+and+minimization+problems%2E+The+annals+of+probability%2C+pages+146%E2%80%93158%2E)
19. Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. 2023. Model merging by uncertainty-based gradient matching. arXiv preprint arXiv:2310.12808. — [arXiv:2310.12808](https://arxiv.org/abs/2310.12808) [find](https://scholar.google.com/scholar?q=Nico+Daheim%2C+Thomas+M%C3%B6llenhoff%2C+Edoardo+Maria+Ponti%2C+Iryna+Gurevych%2C+and+Mohammad+Emtiyaz+Khan%2E+2023%2E+Model+merging+by+uncertainty%2Dbased+gradient+matching%2E+arXiv+preprint+arXiv%3A2310%2E12808%2E+%E2%80%94+%5BarXiv%3A23)
20. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems. [find](https://scholar.google.com/scholar?q=Thomas+G%2E+Dietterich%2E+2000%2E+Ensemble+methods+in+machine+learning%2E+In+Multiple+Classifier+Systems%2E)
21. Olivier Ferret. 2025. Projeter pour mieux fusionner: une histoire de bandit et de lit. In Proc. TALN. [find](https://scholar.google.com/scholar?q=Olivier+Ferret%2E+2025%2E+Projeter+pour+mieux+fusionner%3A+une+histoire+de+bandit+et+de+lit%2E+In+Proc%2E+TALN%2E)
22. Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently Identifying Task Groupings for Multi-Task Learning. In Proc. NEURIPS. [find](https://scholar.google.com/scholar?q=Chris+Fifty%2C+Ehsan+Amid%2C+Zhe+Zhao%2C+Tianhe+Yu%2C+Rohan+Anil%2C+and+Chelsea+Finn%2E+2021%2E+Efficiently+Identifying+Task+Groupings+for+Multi%2DTask+Learning%2E+In+Proc%2E+NEURIPS%2E)
23. Philippe Formont, Maxime Darrin, Banafsheh Karim-ian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, and Pablo Piantanida. 2025. Learning task-agnostic representations through multi-teacher distillation. arXiv preprint arXiv:2510.18680. [find](https://scholar.google.com/scholar?q=Philippe+Formont%2C+Maxime+Darrin%2C+Banafsheh+Karim%2Dian%2C+Jackie+CK+Cheung%2C+Eric+Granger%2C+Ismail+Ben+Ayed%2C+Mohammadhadi+Shateri%2C+and+Pablo+Piantanida%2E+2025%2E+Learning+task%2Dagnostic+representations+through+)
24. Loïc Fosse, Frederic Bechet, Benoit Favre, Géraldine Damnati, Gwénolé Lecorvé, Maxime Darrin, Philippe Formont, and Pablo Piantanida. 2025. Statistical deficiency for task inclusion estimation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Lo%C3%AFc+Fosse%2C+Frederic+Bechet%2C+Benoit+Favre%2C+G%C3%A9raldine+Damnati%2C+Gw%C3%A9nol%C3%A9+Lecorv%C3%A9%2C+Maxime+Darrin%2C+Philippe+Formont%2C+and+Pablo+Piantanida%2E+2025%2E+Statistical+deficiency+for+task+inclusion+estimation%2E+In+Pro)
25. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. [Arcee’s mergekit: A toolkit for merging large language models](https://md.research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/). arXiv preprint arXiv:2403.13257.
26. Karsten Grove and Hermann Karcher. 1973. How to conjugate c 1-close group actions. Mathematische Zeitschrift. [find](https://scholar.google.com/scholar?q=Karsten+Grove+and+Hermann+Karcher%2E+1973%2E+How+to+conjugate+c+1%2Dclose+group+actions%2E+Mathematische+Zeitschrift%2E)
27. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
28. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proc. ICLR.
29. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). In Proc. ICLR.
30. Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. 2016. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1–32. [find](https://scholar.google.com/scholar?q=Andreas+Maurer%2C+Massimiliano+Pontil%2C+and+Bernardino+Romera%2DParedes%2E+2016%2E+The+benefit+of+multitask+representation+learning%2E+Journal+of+Machine+Learning+Research%2C+17%2881%29%3A1%E2%80%9332%2E)
31. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. [Averaging weights leads to wider optima and better generalization](https://md.research.elamur.ai/free/en/papers/izmailov-2018-averaging-weights-leads-wider-optima/). arXiv preprint arXiv:1803.05407.
32. Frank Nielsen. 2020. On a generalization of the jensen– shannon divergence and the jensen–shannon centroid. Entropy, 22(2):221. [find](https://scholar.google.com/scholar?q=Frank+Nielsen%2E+2020%2E+On+a+generalization+of+the+jensen%E2%80%93+shannon+divergence+and+the+jensen%E2%80%93shannon+centroid%2E+Entropy%2C+22%282%29%3A221%2E)
33. Young Kyun Jang, Dat Huynh, Ashish Shah, Wen-Kai Chen, and Ser-Nam Lim. 2024. Spherical linear interpolation and text-anchoring for zero-shot composed image retrieval. In Proc. ECCV. [find](https://scholar.google.com/scholar?q=Young+Kyun+Jang%2C+Dat+Huynh%2C+Ashish+Shah%2C+Wen%2DKai+Chen%2C+and+Ser%2DNam+Lim%2E+2024%2E+Spherical+linear+interpolation+and+text%2Danchoring+for+zero%2Dshot+composed+image+retrieval%2E+In+Proc%2E+ECCV%2E)
34. Wooseong Jeong and Kuk-Jin Yoon. 2025. Selective task group updates for multi-task optimization. arXiv preprint arXiv:2502.11986. — [arXiv:2502.11986](https://arxiv.org/abs/2502.11986) [find](https://scholar.google.com/scholar?q=Wooseong+Jeong+and+Kuk%2DJin+Yoon%2E+2025%2E+Selective+task+group+updates+for+multi%2Dtask+optimization%2E+arXiv+preprint+arXiv%3A2502%2E11986%2E+%E2%80%94+%5BarXiv%3A2502%2E11986%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2502%2E11986%29)
35. Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Xisen+Jin%2C+Xiang+Ren%2C+Daniel+Preotiuc%2DPietro%2C+and+Pengxiang+Cheng%2E+2023%2E+Dataless+knowledge+fusion+by+merging+weights+of+language+models%2E+In+Proc%2E+ICLR%2E)
36. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Peter+Clark%2C+Michal+Guerquin%2C+Peter+Jansen%2C+and+Ashish+Sabharwal%2E+2020%2E+Qasc%3A+A+dataset+for+question+answering+via+sentence+composition%2E+arXiv%3A1910%2E11473v2%2E)
37. Hyoseo Kim, Dongyoon Han, and Junsuk Choe. 2024. Negmerge: Consensual weight negation for strong machine unlearning. arXiv preprint arXiv:2410.05583. — [arXiv:2410.05583](https://arxiv.org/abs/2410.05583) [find](https://scholar.google.com/scholar?q=Hyoseo+Kim%2C+Dongyoon+Han%2C+and+Junsuk+Choe%2E+2024%2E+Negmerge%3A+Consensual+weight+negation+for+strong+machine+unlearning%2E+arXiv+preprint+arXiv%3A2410%2E05583%2E+%E2%80%94+%5BarXiv%3A2410%2E05583%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E05)
38. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
39. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86. [find](https://scholar.google.com/scholar?q=Solomon+Kullback+and+Richard+A+Leibler%2E+1951%2E+On+information+and+sufficiency%2E+The+annals+of+mathematical+statistics%2C+22%281%29%3A79%E2%80%9386%2E)
40. Kevin Kuo, Amrith Setlur, Kartik Srinivas, Aditi Raghunathan, and Virginia Smith. 2025. Exact unlearning of finetuning data via model merging at scale. arXiv preprint arXiv:2504.04626. — [arXiv:2504.04626](https://arxiv.org/abs/2504.04626) [find](https://scholar.google.com/scholar?q=Kevin+Kuo%2C+Amrith+Setlur%2C+Kartik+Srinivas%2C+Aditi+Raghunathan%2C+and+Virginia+Smith%2E+2025%2E+Exact+unlearning+of+finetuning+data+via+model+merging+at+scale%2E+arXiv+preprint+arXiv%3A2504%2E04626%2E+%E2%80%94+%5BarXiv%3A2504%2E0)
41. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Proc. Findings of EMNLP. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Wangchunshu+Zhou%2C+Ming+Shen%2C+Pei+Zhou%2C+Chandra+Bhagavatula%2C+Yejin+Choi%2C+and+Xiang+Ren%2E+2020%2E+CommonGen%3A+A+constrained+text+generation+challenge+for+generative+commonsense+reasoning%2E+I)
42. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Proc. of Text Summarization Branches Out, Barcelona, Spain.
43. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. [Learning word vectors for sentiment analysis](https://md.research.elamur.ai/free/en/papers/maas-2011-learning-word-vectors-sentiment-analysis/). In Proc. ACL.
44. Michael S Matena and Colin A Raffel. 2022. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). In Proc. NEURIPS.
45. Frank Nielsen and Richard Nock. 2007. On the centroids of symmetrized bregman divergences. arXiv preprint arXiv:0711.3242. — [arXiv:0711.3242](https://arxiv.org/abs/0711.3242) [find](https://scholar.google.com/scholar?q=Frank+Nielsen+and+Richard+Nock%2E+2007%2E+On+the+centroids+of+symmetrized+bregman+divergences%2E+arXiv+preprint+arXiv%3A0711%2E3242%2E+%E2%80%94+%5BarXiv%3A0711%2E3242%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F0711%2E3242%29)
46. Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Proc. NEURIPS. [find](https://scholar.google.com/scholar?q=Guillermo+Ortiz%2DJimenez%2C+Alessandro+Favero%2C+and+Pascal+Frossard%2E+2023%2E+Task+arithmetic+in+the+tangent+space%3A+Improved+editing+of+pre%2Dtrained+models%2E+In+Proc%2E+NEURIPS%2E)
47. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. [Adapterfusion: Non-destructive task composition for transfer learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/). In Proc. EACL.
48. Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023. Modular deep learning. arXiv preprint arXiv:2302.11529. — [arXiv:2302.11529](https://arxiv.org/abs/2302.11529) [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Sebastian+Ruder%2C+Ivan+Vulic%CC%81%2C+and+Edoardo+Maria+Ponti%2E+2023%2E+Modular+deep+learning%2E+arXiv+preprint+arXiv%3A2302%2E11529%2E+%E2%80%94+%5BarXiv%3A2302%2E11529%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E11529%29)
49. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
50. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proc. EMNLP.
51. Henry Scheffé. 1947. A Useful Convergence Theorem for Probability Distributions. The Annals of Mathematical Statistics. [find](https://scholar.google.com/scholar?q=Henry+Scheff%C3%A9%2E+1947%2E+A+Useful+Convergence+Theorem+for+Probability+Distributions%2E+The+Annals+of+Mathematical+Statistics%2E)
52. Zhang Shengyu, Dong Linfeng, Li Xiaoya, Zhang Sen, Sun Xiaofei, Wang Shuhe, Li Jiwei, Runyi Hu, Zhang Tianwei, Fei Wu, and 1 others. 2023. [Instruction tuning for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/). arXiv preprint arXiv:2308.10792.
53. Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2020. Which tasks should be learned together in multi-task learning? In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Trevor+Standley%2C+Amir+Zamir%2C+Dawn+Chen%2C+Leonidas+Guibas%2C+Jitendra+Malik%2C+and+Silvio+Savarese%2E+2020%2E+Which+tasks+should+be+learned+together+in+multi%2Dtask+learning%3F+In+Proc%2E+ICLR%2E)
54. Charles Stein. 1956. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proc. of the third Berkeley symposium on mathematical statistics and probability. [find](https://scholar.google.com/scholar?q=Charles+Stein%2E+1956%2E+Inadmissibility+of+the+usual+estimator+for+the+mean+of+a+multivariate+normal+distribution%2E+In+Proc%2E+of+the+third+Berkeley+symposium+on+mathematical+statistics+and+probability%2E)
55. Gilbert W Stewart. 1993. On the early history of the singular value decomposition. SIAM review. [find](https://scholar.google.com/scholar?q=Gilbert+W+Stewart%2E+1993%2E+On+the+early+history+of+the+singular+value+decomposition%2E+SIAM+review%2E)
56. Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. arXiv preprint arXiv:1910.10699. — [arXiv:1910.10699](https://arxiv.org/abs/1910.10699) [find](https://scholar.google.com/scholar?q=Yonglong+Tian%2C+Dilip+Krishnan%2C+and+Phillip+Isola%2E+2019%2E+Contrastive+representation+distillation%2E+arXiv+preprint+arXiv%3A1910%2E10699%2E+%E2%80%94+%5BarXiv%3A1910%2E10699%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E10699%29)
57. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Proc. NEURIPS.
58. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. [Language models are super mario: Absorbing abilities from homologous models as a free lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/). In Proc. ICML.
59. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proc. EMNLP.
60. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. [Gradient surgery for multi-task learning](https://md.research.elamur.ai/free/en/papers/yu-2020-gradient-surgery-multi-task-learning/). In Proc. NEURIPS.
61. Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813. — [arXiv:2405.07813](https://arxiv.org/abs/2405.07813) [find](https://scholar.google.com/scholar?q=Ke+Wang%2C+Nikolaos+Dimitriadis%2C+Guillermo+Ortiz%2DJimenez%2C+Fran%C3%A7ois+Fleuret%2C+and+Pascal+Frossard%2E+2024%2E+Localizing+task+information+for+improved+model+merging+and+compression%2E+arXiv+preprint+arXiv%3A2405%2E0)
62. Yuefeng Zhang. 2023. A rate-distortion-classification approach for lossy image compression. Digital Signal Processing, 141:104163. [find](https://scholar.google.com/scholar?q=Yuefeng+Zhang%2E+2023%2E+A+rate%2Ddistortion%2Dclassification+approach+for+lossy+image+compression%2E+Digital+Signal+Processing%2C+141%3A104163%2E)
63. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771.
64. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. Metagpt: Merging large language models using model exclusive task arithmetic. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Yuyan+Zhou%2C+Liang+Song%2C+Bingning+Wang%2C+and+Weipeng+Chen%2E+2024%2E+Metagpt%3A+Merging+large+language+models+using+model+exclusive+task+arithmetic%2E+In+Proc%2E+EMNLP%2E)
65. Andrew KC Wong and Manlai You. 1985. Entropy and distance of random graphs with application to structural pattern recognition. IEEE transactions on pattern analysis and machine intelligence, (5):599– 609. [find](https://scholar.google.com/scholar?q=Andrew+KC+Wong+and+Manlai+You%2E+1985%2E+Entropy+and+distance+of+random+graphs+with+application+to+structural+pattern+recognition%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+%285%29%3A599%E2%80%93+)
66. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and 1 others. 2022. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/)…
67. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. [Ties-merging: Resolving interference when merging models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/). In Proc. NEURIPS.
68. Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. 2024. What matters for model merging at scale? arXiv preprint arXiv:2410.03617. [find](https://scholar.google.com/scholar?q=Prateek+Yadav%2C+Tu+Vu%2C+Jonathan+Lai%2C+Alexandra+Chronopoulou%2C+Manaal+Faruqui%2C+Mohit+Bansal%2C+and+Tsendsuren+Munkhdalai%2E+2024%2E+What+matters+for+model+merging+at+scale%3F+arXiv+preprint+arXiv%3A2410%2E03617%2E)
69. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024a. Qwen2.5 technical report. arXiv preprint arX… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
70. Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024b. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666. [find](https://scholar.google.com/scholar?q=Enneng+Yang%2C+Li+Shen%2C+Guibing+Guo%2C+Xingwei+Wang%2C+Xiaochun+Cao%2C+Jie+Zhang%2C+and+Dacheng+Tao%2E+2024b%2E+Model+merging+in+llms%2C+mllms%2C+and+beyond%3A+Methods%2C+theories%2C+applications+and+opportunities%2E+arXiv+pre)
71. Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024c. Adamerging: Adaptive model merging for multi-task learning. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Enneng+Yang%2C+Zhenyi+Wang%2C+Li+Shen%2C+Shiwei+Liu%2C+Guibing+Guo%2C+Xingwei+Wang%2C+and+Dacheng+Tao%2E+2024c%2E+Adamerging%3A+Adaptive+model+merging+for+multi%2Dtask+learning%2E+In+Proc%2E+ICLR%2E)

*Updated 1 November 2026*