---
title: "Merging Models with Fisher-Weighted Averaging — Matena 2021"
description: "Merging, or combining models by averaging their parameters, works only for networks with a shared architecture and initialization."
authors: ["Michael Matena", "Colin Raffel"]
year: 2021
venue: "arXiv:2111.09832"
source: "https://arxiv.org/abs/2111.09832"
doi: "10.48550/arxiv.2111.09832"
arxiv: "2111.09832"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Merging Models with Fisher-Weighted Averaging

# Merging Models with Fisher-Weighted Averaging

**Authors:** Michael Matena, [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/) · **Year:** 2021 · **Venue:** arXiv:2111.09832 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2111.09832) · [PDF](https://arxiv.org/pdf/2111.09832) · [DOI 10.48550/arxiv.2111.09832](https://doi.org/10.48550/arxiv.2111.09832) · [arXiv 2111.09832](https://arxiv.org/abs/2111.09832) · [OpenAlex](https://openalex.org/W3214411122)

**Citations:** 46 citations (openalex-snapshot · 2026-10-14) · [144](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Merging, or combining models by averaging their parameters, works only for networks with a shared architecture and initialization. It already appears in ensembling, federated learning and robust fine-tuning, but it treats every weight as equally important. Matena and Raffel reframe merging as a search for parameters that come close to jointly maximizing the likelihood under each model's posterior. Under this view, plain averaging implicitly assumes an isotropic Gaussian posterior. They replace it with a Laplace approximation that uses each model's diagonal Fisher information as the precision. The merged weight then becomes a per-parameter average weighted by Fisher values, with optional model-level coefficients. They tested this "Fisher merging" on ensembling, robust fine-tuning, intermediate-task transfer and domain adaptation with BERT and RoBERTa models. It beats unweighted averaging in most settings. It also matches, and occasionally exceeds, gradient-based transfer learning at a fraction of the compute. This matters because capabilities from several checkpoints can be combined in closed form without retraining, and the code is publicly released.

## Key points
- **Method:** each merged parameter is a weighted average of the corresponding parameters from the source models. The weights are diagonal Fisher values estimated from per-example gradients on training data. Scalar λ coefficients set how much each model contributes.
- **Domain adaptation setup:** RoBERTa-base checkpoints were merged with a domain-adaptive pre-trained (DAPT) checkpoint, following Gururangan et al. The tasks were ChemProt (BioMed domain, micro-F1) and ACL-ARC and SciERC (CS domain, macro-F1).
- **ACL-ARC results:** this task showed the largest gain. Fisher merging scored 73.2, compared with 70.5 for the unmerged model and 71.5 for fine-tuning from DAPT.
- **ChemProt and SciERC results:** gains were small. On ChemProt, Fisher merging scored 83.1 versus 82.7 unmerged. On SciERC, isotropic merging (81.7) slightly edged out Fisher merging (81.3).
- **Code:** released at github.com/mmatena/model_merging (arXiv:2111.09832).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

144 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 137 not yet.

- [Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/anikina-2025-reverse-probing-evaluating-knowledge-transfer/)
- [StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements](https://md.research.elamur.ai/free/en/papers/fisher-2024-styleremix-interpretable-authorship-obfuscation-distillation/)
- [Editing Models with Task Arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/)
- [TIES-Merging: Resolving Interference When Merging Models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{matena2021merging,
  title = {{Merging Models with Fisher-Weighted Averaging}},
  author = {Matena, Michael and Raffel, Colin},
  year = {2021},
  doi = {10.48550/arxiv.2111.09832},
  eprint = {2111.09832},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.09832}
}
```

- [← previous MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/zhu-2021-mediasum-large-scale-media-interview/)
- [next → Metacommunicative-why fragments as probes into the grammar of the speech act layer](https://md.research.elamur.ai/free/en/papers/woods-2021-metacommunicative-i-why-i-fragments/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

68 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., Soatto, S., and Perona, P. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6430–6439, 2019. [find](https://scholar.google.com/scholar?q=Achille%2C+A%2E%2C+Lam%2C+M%2E%2C+Tewari%2C+R%2E%2C+Ravichandran%2C+A%2E%2C+Maji%2C+S%2E%2C+Fowlkes%2C+C%2E+C%2E%2C+Soatto%2C+S%2E%2C+and+Perona%2C+P%2E+Task2vec%3A+Task+embedding+for+meta%2Dlearning%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Confer)
2. Achille, A., Paolini, G., and Soatto, S. Where is the information in a deep neural network? arXiv preprint arXiv:1905.12213, 2019. — [arXiv:1905.12213](https://arxiv.org/abs/1905.12213) [find](https://scholar.google.com/scholar?q=Achille%2C+A%2E%2C+Paolini%2C+G%2E%2C+and+Soatto%2C+S%2E+Where+is+the+information+in+a+deep+neural+network%3F+arXiv+preprint+arXiv%3A1905%2E12213%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E12213%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1905%2E12213%29)
3. Amari, S. Neural learning in structured parameter spaces-natural riemannian gradient. Advances in neural information processing systems, pp. 127–133, 1997. [find](https://scholar.google.com/scholar?q=Amari%2C+S%2E+Neural+learning+in+structured+parameter+spaces%2Dnatural+riemannian+gradient%2E+Advances+in+neural+information+processing+systems%2C+pp%2E+127%E2%80%93133%2C+1997%2E)
4. Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019. [find](https://scholar.google.com/scholar?q=Barbu%2C+A%2E%2C+Mayo%2C+D%2E%2C+Alverio%2C+J%2E%2C+Luo%2C+W%2E%2C+Wang%2C+C%2E%2C+Gutfreund%2C+D%2E%2C+Tenenbaum%2C+J%2E%2C+and+Katz%2C+B%2E+Objectnet%3A+A+large%2Dscale+bias%2Dcontrolled+dataset+for+pushing+the+limits+of+object+recognition+models%2E+In)
5. Bartels, L. M. Specification uncertainty and model averaging. American Journal of Political Science, 1997. [find](https://scholar.google.com/scholar?q=Bartels%2C+L%2E+M%2E+Specification+uncertainty+and+model+averaging%2E+American+Journal+of+Political+Science%2C+1997%2E)
6. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015. [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E%2C+Angeli%2C+G%2E%2C+Potts%2C+C%2E%2C+and+Manning%2C+C%2E+D%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+arXiv+preprint+arXiv%3A1508%2E05326%2C+2015%2E)
7. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. [find](https://scholar.google.com/scholar?q=Cer%2C+D%2E%2C+Diab%2C+M%2E%2C+Agirre%2C+E%2E%2C+Lopez%2DGazpio%2C+I%2E%2C+and+Specia%2C+L%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity%2Dmultilingual+and+cross%2Dlingual+focused+evaluation%2E+arXiv+preprint+arXiv%3A1708%2E00055%2C+201)
8. Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177–190. Springer, 2005. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+and+Magnini%2C+B%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+pp%2E+177%E2%80%93190%2E+Springer%2C+2005%2E)
9. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In Advances in neural information processing systems, 2015. [find](https://scholar.google.com/scholar?q=Dai%2C+A%2E+M%2E+and+Le%2C+Q%2E+V%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+2015%2E)
10. Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems (NeurIPS), 2021. [find](https://scholar.google.com/scholar?q=Daxberger%2C+E%2E%2C+Kristiadi%2C+A%2E%2C+Immer%2C+A%2E%2C+Eschenhagen%2C+R%2E%2C+Bauer%2C+M%2E%2C+and+Hennig%2C+P%2E+Laplace+redux%2Deffortless+bayesian+deep+learning%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2021%2E)
11. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009. [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E%2C+Dong%2C+W%2E%2C+Socher%2C+R%2E%2C+Li%2C+L%2E%2DJ%2E%2C+Li%2C+K%2E%2C+and+Fei%2DFei%2C+L%2E+ImageNet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+2009%2E)
12. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
13. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
14. Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. [find](https://scholar.google.com/scholar?q=Dolan%2C+W%2E+B%2E+and+Brockett%2C+C%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2C+2005%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
15. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [find](https://scholar.google.com/scholar?q=Dosovitskiy%2C+A%2E%2C+Beyer%2C+L%2E%2C+Kolesnikov%2C+A%2E%2C+Weissenborn%2C+D%2E%2C+Zhai%2C+X%2E%2C+Unterthiner%2C+T%2E%2C+Dehghani%2C+M%2E%2C+Minderer%2C+M%2E%2C+Heigold%2C+G%2E%2C+Gelly%2C+S%2E%2C+Uszkoreit%2C+J%2E%2C+and+Houlsby%2C+N%2E+An+image+is+worth+16x16+words)
16. Fisher, R. A. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222(594-604):309–368, 1922. [find](https://scholar.google.com/scholar?q=Fisher%2C+R%2E+A%2E+On+the+mathematical+foundations+of+theoretical+statistics%2E+Philosophical+Transactions+of+the+Royal+Society+of+London%2E+Series+A%2C+Containing+Papers+of+a+Mathematical+or+Physical+Character%2C)
17. Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013. [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2E+J%2E%2C+Mirza%2C+M%2E%2C+Xiao%2C+D%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+An+empirical+investigation+of+catastrophic+forgetting+in+gradient%2Dbased+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E6211%2C+2013%2E)
18. Grosse, R. and Martens, J. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pp. 573–582. PMLR, 2016. [find](https://scholar.google.com/scholar?q=Grosse%2C+R%2E+and+Martens%2C+J%2E+A+kronecker%2Dfactored+approximate+fisher+matrix+for+convolution+layers%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+573%E2%80%93582%2E+PMLR%2C+2016%2E)
19. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%2C+Marasovic%CC%81%2C+A%2E%2C+Swayamdipta%2C+S%2E%2C+Lo%2C+K%2E%2C+Beltagy%2C+I%2E%2C+Downey%2C+D%2E%2C+and+Smith%2C+N%2E+A%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+arXiv+preprint+arXiv%3A2004%2E10964%2C+)
20. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. International Conference on Computer Vision (ICCV), 202… [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Basart%2C+S%2E%2C+Mu%2C+N%2E%2C+Kadavath%2C+S%2E%2C+Wang%2C+F%2E%2C+Dorundo%2C+E%2E%2C+Desai%2C+R%2E%2C+Zhu%2C+T%2E%2C+Parajuli%2C+S%2E%2C+Guo%2C+M%2E%2C+Song%2C+D%2E%2C+Steinhardt%2C+J%2E%2C+and+Gilmer%2C+J%2E+The+many+faces+of+robustness%3A+A+critical+ana)
21. Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. Conference on Computer Vision and Pattern Recognition (CVPR), 2021. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Zhao%2C+K%2E%2C+Basart%2C+S%2E%2C+Steinhardt%2C+J%2E%2C+and+Song%2C+D%2E+Natural+adversarial+examples%2E+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2021%2E)
22. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E%2C+Vinyals%2C+O%2E%2C+and+Dean%2C+J%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
23. Iyer, S., Dandekar, N., and Csernai, K. First quora dataset release: Question pairs, 2017. URL [https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs). [↗](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs)
24. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, 2014. [find](https://scholar.google.com/scholar?q=Jia%2C+Y%2E%2C+Shelhamer%2C+E%2E%2C+Donahue%2C+J%2E%2C+Karayev%2C+S%2E%2C+Long%2C+J%2E%2C+Girshick%2C+R%2E%2C+Guadarrama%2C+S%2E%2C+and+Darrell%2C+T%2E+Caffe%3A+Convolutional+architecture+for+fast+feature+embedding%2E+In+Proceedings+of+the+22nd+ACM+i) [our card](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/)
25. Jurgens, D., Kumar, S., Hoover, R., McFarland, D., and Jurafsky, D. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406, 2018. [find](https://scholar.google.com/scholar?q=Jurgens%2C+D%2E%2C+Kumar%2C+S%2E%2C+Hoover%2C+R%2E%2C+McFarland%2C+D%2E%2C+and+Jurafsky%2C+D%2E+Measuring+the+evolution+of+a+scientific+field+through+citation+frames%2E+Transactions+of+the+Association+for+Computational+Linguistics) [our card](https://md.research.elamur.ai/free/en/papers/jurgens-2018-measuring-evolution-scientific-field-through/)
26. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [find](https://scholar.google.com/scholar?q=Kaplan%2C+J%2E%2C+McCandlish%2C+S%2E%2C+Henighan%2C+T%2E%2C+Brown%2C+T%2E+B%2E%2C+Chess%2C+B%2E%2C+Child%2C+R%2E%2C+Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+and+Amodei%2C+D%2E+Scaling+laws+for+neural+language+models%2E+arXiv+preprint+arXiv%3A2001%2E08361%2C+20)
27. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Ba%2C+J%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
28. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. [find](https://scholar.google.com/scholar?q=Kirkpatrick%2C+J%2E%2C+Pascanu%2C+R%2E%2C+Rabinowitz%2C+N%2E%2C+Veness%2C+J%2E%2C+Desjardins%2C+G%2E%2C+Rusu%2C+A%2E+A%2E%2C+Milan%2C+K%2E%2C+Quan%2C+J%2E%2C+Ramalho%2C+T%2E%2C+Grabska%2DBarwinska%2C+A%2E%2C+et+al%2E+Overcoming+catastrophic+forgetting+in+neural+netw)
29. Kocijan, V., Cretu, A.-M., Camburu, O.-M., Yordanov, Y., and Lukasiewicz, T. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290, 2019. — [arXiv:1905.06290](https://arxiv.org/abs/1905.06290) [find](https://scholar.google.com/scholar?q=Kocijan%2C+V%2E%2C+Cretu%2C+A%2E%2DM%2E%2C+Camburu%2C+O%2E%2DM%2E%2C+Yordanov%2C+Y%2E%2C+and+Lukasiewicz%2C+T%2E+A+surprisingly+robust+trick+for+winograd+schema+challenge%2E+arXiv+preprint+arXiv%3A1905%2E06290%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E06290%5D%28http)
30. Kringelum, J., Kjaerulff, S. K., Brunak, S., Lund, O., Oprea, T. I., and Taboureau, O. Chemprot-3.0: a global chemical biology diseases mapping. Database, 2016, 2016. [find](https://scholar.google.com/scholar?q=Kringelum%2C+J%2E%2C+Kjaerulff%2C+S%2E+K%2E%2C+Brunak%2C+S%2E%2C+Lund%2C+O%2E%2C+Oprea%2C+T%2E+I%2E%2C+and+Taboureau%2C+O%2E+Chemprot%2D3%2E0%3A+a+global+chemical+biology+diseases+mapping%2E+Database%2C+2016%2C+2016%2E)
31. Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. [find](https://scholar.google.com/scholar?q=Levesque%2C+H%2E%2C+Davis%2C+E%2E%2C+and+Morgenstern%2C+L%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2C+2012%2E)
32. Liu, X., Masana, M., Herranz, L., Van de Weijer, J., Lopez, A. M., and Bagdanov, A. D. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2262–2268. IEEE, 2018. [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Masana%2C+M%2E%2C+Herranz%2C+L%2E%2C+Van+de+Weijer%2C+J%2E%2C+Lopez%2C+A%2E+M%2E%2C+and+Bagdanov%2C+A%2E+D%2E+Rotate+your+networks%3A+Better+weight+consolidation+and+less+catastrophic+forgetting%2E+In+2018+24th+International+Co)
33. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Ott%2C+M%2E%2C+Goyal%2C+N%2E%2C+Du%2C+J%2E%2C+Joshi%2C+M%2E%2C+Chen%2C+D%2E%2C+Levy%2C+O%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+preprint+arXiv%3A1907%2E1169)
34. Lo, K., Wang, L. L., Neumann, M., Kinney, R., and Weld, D. S. S2orc: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782, 2019. [find](https://scholar.google.com/scholar?q=Lo%2C+K%2E%2C+Wang%2C+L%2E+L%2E%2C+Neumann%2C+M%2E%2C+Kinney%2C+R%2E%2C+and+Weld%2C+D%2E+S%2E+S2orc%3A+The+semantic+scholar+open+research+corpus%2E+arXiv+preprint+arXiv%3A1911%2E02782%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/lo-2019-s2orc-semantic-scholar-open-research/)
35. Luan, Y., He, L., Ostendorf, M., and Hajishirzi, H. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv:1808.09602, 2018. [find](https://scholar.google.com/scholar?q=Luan%2C+Y%2E%2C+He%2C+L%2E%2C+Ostendorf%2C+M%2E%2C+and+Hajishirzi%2C+H%2E+Multi%2Dtask+identification+of+entities%2C+relations%2C+and+coreference+for+scientific+knowledge+graph+construction%2E+arXiv+preprint+arXiv%3A1808%2E09602%2C+2018) [our card](https://md.research.elamur.ai/free/en/papers/luan-2018-multi-task-identification-entities-relations/)
36. MacKay, D. J. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448–472, 1992. [find](https://scholar.google.com/scholar?q=MacKay%2C+D%2E+J%2E+A+practical+bayesian+framework+for+backpropagation+networks%2E+Neural+computation%2C+4%283%29%3A448%E2%80%93472%2C+1992%2E)
37. Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408–2417. PMLR, 2015. [find](https://scholar.google.com/scholar?q=Martens%2C+J%2E+and+Grosse%2C+R%2E+Optimizing+neural+networks+with+kronecker%2Dfactored+approximate+curvature%2E+In+International+conference+on+machine+learning%2C+pp%2E+2408%E2%80%932417%2E+PMLR%2C+2015%2E)
38. Martens, J., Ba, J., and Johnson, M. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Martens%2C+J%2E%2C+Ba%2C+J%2E%2C+and+Johnson%2C+M%2E+Kronecker%2Dfactored+curvature+approximations+for+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E)
39. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273–1282. PMLR, 2017. [find](https://scholar.google.com/scholar?q=McMahan%2C+B%2E%2C+Moore%2C+E%2E%2C+Ramage%2C+D%2E%2C+Hampson%2C+S%2E%2C+and+y+Arcas%2C+B%2E+A%2E+Communication%2Defficient+learning+of+deep+networks+from+decentralized+data%2E+In+Artificial+Intelligence+and+Statistics%2C+pp%2E+1273%E2%80%931282%2E)
40. Nielsen, F. An elementary introduction to information geometry. Entropy, 22(10):1100, 2020. [find](https://scholar.google.com/scholar?q=Nielsen%2C+F%2E+An+elementary+introduction+to+information+geometry%2E+Entropy%2C+22%2810%29%3A1100%2C+2020%2E)
41. Nielsen, F. and Nock, R. Sided and symmetrized bregman centroids. IEEE transactions on Information Theory, 55(6), 2009. [find](https://scholar.google.com/scholar?q=Nielsen%2C+F%2E+and+Nock%2C+R%2E+Sided+and+symmetrized+bregman+centroids%2E+IEEE+transactions+on+Information+Theory%2C+55%286%29%2C+2009%2E)
42. Opitz, D. W. and Shavlik, J. W. Actively searching for an effective neural network ensemble. Connection Science, 8(3-4):337–354, 1996. [find](https://scholar.google.com/scholar?q=Opitz%2C+D%2E+W%2E+and+Shavlik%2C+J%2E+W%2E+Actively+searching+for+an+effective+neural+network+ensemble%2E+Connection+Science%2C+8%283%2D4%29%3A337%E2%80%93354%2C+1996%2E)
43. Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014. [find](https://scholar.google.com/scholar?q=Oquab%2C+M%2E%2C+Bottou%2C+L%2E%2C+Laptev%2C+I%2E%2C+and+Sivic%2C+J%2E+Learning+and+transferring+mid%2Dlevel+image+representations+using+convolutional+neural+networks%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision)
44. Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009. [find](https://scholar.google.com/scholar?q=Pan%2C+S%2E+J%2E+and+Yang%2C+Q%2E+A+survey+on+transfer+learning%2E+IEEE+Transactions+on+knowledge+and+data+engineering%2C+22%2810%29%3A1345%E2%80%931359%2C+2009%2E)
45. Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) [find](https://scholar.google.com/scholar?q=Pascanu%2C+R%2E+and+Bengio%2C+Y%2E+Revisiting+natural+gradient+for+deep+networks%2E+arXiv+preprint+arXiv%3A1301%2E3584%2C+2013%2E+%E2%80%94+%5BarXiv%3A1301%2E3584%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1301%2E3584%29)
46. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2C+2018%2E)
47. Phang, J., Févry, T., and Bowman, S. R. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. [find](https://scholar.google.com/scholar?q=Phang%2C+J%2E%2C+F%C3%A9vry%2C+T%2E%2C+and+Bowman%2C+S%2E+R%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
48. Phang, J., Htut, P. M., Pruksachatkun, Y., Liu, H., Vania, C., Kann, K., Calixto, I., and Bowman, S. R. English intermediate-task training improves zero-shot cross-lingual transfer too. arXiv preprint arXiv:2005.13013, 2020. [find](https://scholar.google.com/scholar?q=Phang%2C+J%2E%2C+Htut%2C+P%2E+M%2E%2C+Pruksachatkun%2C+Y%2E%2C+Liu%2C+H%2E%2C+Vania%2C+C%2E%2C+Kann%2C+K%2E%2C+Calixto%2C+I%2E%2C+and+Bowman%2C+S%2E+R%2E+English+intermediate%2Dtask+training+improves+zero%2Dshot+cross%2Dlingual+transfer+too%2E+arXiv+preprint)
49. Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992. [find](https://scholar.google.com/scholar?q=Polyak%2C+B%2E+T%2E+and+Juditsky%2C+A%2E+B%2E+Acceleration+of+stochastic+approximation+by+averaging%2E+SIAM+journal+on+control+and+optimization%2C+30%284%29%3A838%E2%80%93855%2C+1992%2E)
50. Posada, D. and Buckley, T. R. Model selection and model averaging in phylogenetics: advantages of akaike information criterion and bayesian approaches over likelihood ratio tests. Systematic biology, 53(5), 2004. [find](https://scholar.google.com/scholar?q=Posada%2C+D%2E+and+Buckley%2C+T%2E+R%2E+Model+selection+and+model+averaging+in+phylogenetics%3A+advantages+of+akaike+information+criterion+and+bayesian+approaches+over+likelihood+ratio+tests%2E+Systematic+biology%2C+)
51. Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., and Bowman, S. R. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint arXiv:2005.00628, 2020. — [arXiv:2005.00628](https://arxiv.org/abs/2005.00628) [find](https://scholar.google.com/scholar?q=Pruksachatkun%2C+Y%2E%2C+Phang%2C+J%2E%2C+Liu%2C+H%2E%2C+Htut%2C+P%2E+M%2E%2C+Zhang%2C+X%2E%2C+Pang%2C+R%2E+Y%2E%2C+Vania%2C+C%2E%2C+Kann%2C+K%2E%2C+and+Bowman%2C+S%2E+R%2E+Intermediate%2Dtask+transfer+learning+with+pretrained+models+for+natural+language+under)
52. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training, 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Narasimhan%2C+K%2E%2C+Salimans%2C+T%2E%2C+and+Sutskever%2C+I%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2C+2018%2E)
53. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+arXiv+preprin)
54. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Zhang%2C+J%2E%2C+Lopyrev%2C+K%2E%2C+and+Liang%2C+P%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
55. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Jia%2C+R%2E%2C+and+Liang%2C+P%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+squad%2E+arXiv+preprint+arXiv%3A1806%2E03822%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
56. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning (ICML), 2019. [find](https://scholar.google.com/scholar?q=Recht%2C+B%2E%2C+Roelofs%2C+R%2E%2C+Schmidt%2C+L%2E%2C+and+Shankar%2C+V%2E+Do+ImageNet+classifiers+generalize+to+ImageNet%3F+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2019%2E)
57. Ruder, S., Peters, M. E., Swayamdipta, S., and Wolf, T. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15–18, 2019. [find](https://scholar.google.com/scholar?q=Ruder%2C+S%2E%2C+Peters%2C+M%2E+E%2E%2C+Swayamdipta%2C+S%2E%2C+and+Wolf%2C+T%2E+Transfer+learning+in+natural+language+processing%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)
58. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 2015. [find](https://scholar.google.com/scholar?q=Russakovsky%2C+O%2E%2C+Deng%2C+J%2E%2C+Su%2C+H%2E%2C+Krause%2C+J%2E%2C+Satheesh%2C+S%2E%2C+Ma%2C+S%2E%2C+Huang%2C+Z%2E%2C+Karpathy%2C+A%2E%2C+Khosla%2C+A%2E%2C+Bernstein%2C+M%2E%2C+et+al%2E+ImageNet+large+scale+visual+recognition+challenge%2E+International+journal) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
59. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Perelygin%2C+A%2E%2C+Wu%2C+J%2E%2C+Chuang%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+Ng%2C+A%2E+Y%2E%2C+and+Potts%2C+C%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Proceedings+of+the+2013+confer)
60. Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., and Iyyer, M. Exploring and predicting transferability across nlp tasks. arXiv preprint arXiv:2005.00770, 2020. [find](https://scholar.google.com/scholar?q=Vu%2C+T%2E%2C+Wang%2C+T%2E%2C+Munkhdalai%2C+T%2E%2C+Sordoni%2C+A%2E%2C+Trischler%2C+A%2E%2C+Mattarella%2DMicke%2C+A%2E%2C+Maji%2C+S%2E%2C+and+Iyyer%2C+M%2E+Exploring+and+predicting+transferability+across+nlp+tasks%2E+arXiv+preprint+arXiv%3A2005%2E00770%2C+) [our card](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
61. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%2C+Singh%2C+A%2E%2C+Michael%2C+J%2E%2C+Hill%2C+F%2E%2C+Levy%2C+O%2E%2C+and+Bowman%2C+S%2E+R%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1804%2E07461%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
62. Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. [find](https://scholar.google.com/scholar?q=Wang%2C+H%2E%2C+Ge%2C+S%2E%2C+Lipton%2C+Z%2E%2C+and+Xing%2C+E%2E+P%2E+Learning+robust+global+representations+by+penalizing+local+predictive+power%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2019%2E)
63. Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440, 2020. — [arXiv:2002.06440](https://arxiv.org/abs/2002.06440) [find](https://scholar.google.com/scholar?q=Wang%2C+H%2E%2C+Yurochkin%2C+M%2E%2C+Sun%2C+Y%2E%2C+Papailiopoulos%2C+D%2E%2C+and+Khazaeni%2C+Y%2E+Federated+learning+with+matched+averaging%2E+arXiv+preprint+arXiv%3A2002%2E06440%2C+2020%2E+%E2%80%94+%5BarXiv%3A2002%2E06440%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002)
64. Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641, 2019. [find](https://scholar.google.com/scholar?q=Warstadt%2C+A%2E%2C+Singh%2C+A%2E%2C+and+Bowman%2C+S%2E+R%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2C+2019%2E)
65. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. [find](https://scholar.google.com/scholar?q=Wolf%2C+T%2E%2C+Debut%2C+L%2E%2C+Sanh%2C+V%2E%2C+Chaumond%2C+J%2E%2C+Delangue%2C+C%2E%2C+Moi%2C+A%2E%2C+Cistac%2C+P%2E%2C+Rault%2C+T%2E%2C+Louf%2C+R%2E%2C+Funtowicz%2C+M%2E%2C+et+al%2E+Huggingface%E2%80%99s+transformers%3A+State%2Dof%2Dthe%2Dart+natural+language+processing%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
66. Wortsman, M., Ilharco, G., Li, M., Kim, J. W., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01903, 2021. — [arXiv:2109.01903](https://arxiv.org/abs/2109.01903) [find](https://scholar.google.com/scholar?q=Wortsman%2C+M%2E%2C+Ilharco%2C+G%2E%2C+Li%2C+M%2E%2C+Kim%2C+J%2E+W%2E%2C+Hajishirzi%2C+H%2E%2C+Farhadi%2C+A%2E%2C+Namkoong%2C+H%2E%2C+and+Schmidt%2C+L%2E+Robust+fine%2Dtuning+of+zero%2Dshot+models%2E+arXiv+preprint+arXiv%3A2109%2E01903%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E0)
67. Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.05… [find](https://scholar.google.com/scholar?q=Wortsman%2C+M%2E%2C+Ilharco%2C+G%2E%2C+Gadre%2C+S%2E+Y%2E%2C+Roelofs%2C+R%2E%2C+Gontijo%2DLopes%2C+R%2E%2C+Morcos%2C+A%2E+S%2E%2C+Namkoong%2C+H%2E%2C+Farhadi%2C+A%2E%2C+Carmon%2C+Y%2E%2C+Kornblith%2C+S%2E%2C+and+Schmidt%2C+L%2E+Model+soups%3A+averaging+weights+of+multiple) [our card](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/)
68. Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Advances in neural information processing systems, 2014. [find](https://scholar.google.com/scholar?q=Yosinski%2C+J%2E%2C+Clune%2C+J%2E%2C+Bengio%2C+Y%2E%2C+and+Lipson%2C+H%2E+How+transferable+are+features+in+deep+neural+networks%3F+In+Advances+in+neural+information+processing+systems%2C+2014%2E)