---
title: "Reproducible scaling laws for contrastive language-image learning — Cherti 2022"
description: "This paper builds the first systematic, fully reproducible scaling study for contrastive language-image… 84 references, cited by 99 papers here."
authors: ["Mehdi Cherti", "Romain Beaumont", "Ross Wightman", "Mitchell Wortsman", "Gabriel Ilharco", "Cade Gordon", "Christoph Schuhmann", "Ludwig Schmidt", "Jenia Jitsev"]
year: 2022
venue: "arXiv:2212.07143 · Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2818-2829"
source: "https://arxiv.org/abs/2212.07143"
doi: "10.1109/cvpr52729.2023.00276"
arxiv: "2212.07143"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Reproducible scaling laws for contrastive language-image learning

# Reproducible scaling laws for contrastive language-image learning

**Authors:** [Mehdi Cherti](https://md.research.elamur.ai/free/en/authors/mehdi-cherti/), [Romain Beaumont](https://md.research.elamur.ai/free/en/authors/romain-beaumont/), [Ross Wightman](https://md.research.elamur.ai/free/en/authors/ross-wightman/), [Mitchell Wortsman](https://md.research.elamur.ai/free/en/authors/mitchell-wortsman/), [Gabriel Ilharco](https://md.research.elamur.ai/free/en/authors/gabriel-ilharco/), [Cade Gordon](https://md.research.elamur.ai/free/en/authors/cade-gordon/), [Christoph Schuhmann](https://md.research.elamur.ai/free/en/authors/christoph-schuhmann/), [Ludwig Schmidt](https://md.research.elamur.ai/free/en/authors/ludwig-schmidt/) and [Jenia Jitsev](https://md.research.elamur.ai/free/en/authors/jenia-jitsev/) · **Year:** 2022 · **Venue:** [arXiv:2212.07143 · Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2818-2829](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1109/cvpr52729.2023.00276) · [PDF](https://arxiv.org/pdf/2212.07143) · [DOI 10.1109/cvpr52729.2023.00276](https://doi.org/10.1109/cvpr52729.2023.00276) · [arXiv 2212.07143](https://arxiv.org/abs/2212.07143) · [OpenAlex](https://openalex.org/W4386065512)

**Citations:** 651 citations (OpenAlex, 2026-10-14) · [99](https://md.research.elamur.ai/free/en/papers/cherti-2022-reproducible-scaling-laws-contrastive-language/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Earlier scaling-law research mostly used closed data and models, or looked only at language or only at vision. This paper builds the first systematic, fully reproducible scaling study for contrastive language-image pre-training (CLIP). The authors trained CLIP models with the open-source OpenCLIP codebase on public LAION data (subsets of LAION-5B, up to LAION-2B). They varied three factors: vision encoder size (ViT-B/32 up to ViT-g/14), dataset size, and number of samples seen.

They evaluated zero-shot classification, retrieval, linear probing and end-to-end fine-tuning. For each task, they fitted power laws to the best models in each total-compute bin. Performance rose predictably with scale on every task.

The scaling rate depended on the training data. OpenAI's CLIP, trained on private WIT-400M, scaled faster on ImageNet classification. LAION-trained models scaled faster on retrieval. The authors argue that dataset design shapes how models scale on each task, so new datasets should be tested on many tasks. The authors made every trained model publicly available, among them the biggest openly accessible CLIP models of that period, along with the evaluation pipeline.

## Key points
- **Training setup:** subsets LAION-80M, LAION-400M and LAION-2B; samples seen of 3B, 13B or 34B; architectures ViT-B/32, B/16, L/14, H/14 and g/14.
- **Zero-shot results (224 px):** OpenCLIP ViT-H/14 on LAION-2B reached 78.0% on ImageNet, 56.4% average over 35 VTAB+ datasets, and 73.4 MS-COCO image-retrieval Recall@5. OpenAI CLIP L/14 scored 75.5, 55.8 and 61.1 on the same benchmarks.
- **Task-dependent scaling:** WIT-trained models showed steeper scaling on ImageNet and its robustness variants (ImageNet-V2, -R, -Sketch, ObjectNet, -A). LAION-trained models showed steeper scaling on MS-COCO and Flickr30K retrieval.
- **Proposed explanation:** batch-size experiments ruled out the main difference in training recipe. The authors instead point to data curation: LAION was filtered using similarity scores from OpenAI's ViT-B/32 model, which may favour retrieval-style tasks.
- **Extrapolation:** the fitted laws predict about 81.9% zero-shot ImageNet top-1 accuracy for a ViT-G/14 trained from scratch on 68B samples.
- **Limitations:** the scale space is sampled sparsely, and full hyperparameter tuning was not possible at large scale.
- **Code:** github.com/LAION-AI/scaling-laws-openclip

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 206 works · 158,423 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 107 works · 205,102 citations
- [Hugging Face](https://md.research.elamur.ai/free/en/companies/hugging-face/) — Company, New York, United States · 10 works · 19,878 citations
- [Forschungszentrum Jülich](https://md.research.elamur.ai/free/en/labs/forschungszentrum-julich/) — Research institute, Jülich, Germany · 3 works · 1,194 citations

Also on the byline: [Jülich Supercomputing Centre](https://ror.org/04g2j9721).

💡 Concepts (1): [📈 Scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/) · 30 works

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 6,918 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 6,761 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 974 works

## Appears in

Agglomerations: [🏙️ Heerlen](https://md.research.elamur.ai/free/en/agglomerations/aachen/)

## Cite

Cite as: Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt and Jenia Jitsev. 2022. Reproducible scaling laws for contrastive language-image learning. arXiv:2212.07143 · Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2818-2829.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cherti2022reproducible,
  title = {{Reproducible scaling laws for contrastive language-image learning}},
  author = {Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt and Jenia Jitsev},
  year = {2022},
  doi = {10.1109/cvpr52729.2023.00276},
  eprint = {2212.07143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/cvpr52729.2023.00276}
}
```

- [← previous UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://md.research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/)
- [next → ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

84 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Common Crawl. [https://commoncrawl.org](https://commoncrawl.org). 5, 20 [↗](https://commoncrawl.org)
2. Stability AI HPC facility, [https://hpc.stability.ai](https://hpc.stability.ai). 6 [↗](https://hpc.stability.ai)
3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 4 [our card](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
4. Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 9, 26 — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) [find](https://scholar.google.com/scholar?q=Hangbo+Bao%2C+Li+Dong%2C+and+Furu+Wei%2E+Beit%3A+Bert+pre%2Dtraining+of+image+transformers%2E+arXiv+preprint+arXiv%3A2106%2E08254%2C+2021%2E+9%2C+26+%E2%80%94+%5BarXiv%3A2106%2E08254%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E08254%29)
5. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 3, 6, 9 [find](https://scholar.google.com/scholar?q=Andrei+Barbu%2C+David+Mayo%2C+Julian+Alverio%2C+William+Luo%2C+Christopher+Wang%2C+Dan+Gutfreund%2C+Josh+Tenenbaum%2C+and+Boris+Katz%2E+Objectnet%3A+A+large%2Dscale+bias%2Dcontrolled+dataset+for+pushing+the+limits+of+objec)
6. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences of the United States of America, 116:15849–15854, Aug. 2019. 4 [find](https://scholar.google.com/scholar?q=Mikhail+Belkin%2C+Daniel+Hsu%2C+Siyuan+Ma%2C+and+Soumik+Mandal%2E+Reconciling+modern+machine%2Dlearning+practice+and+the+classical+bias%2Dvariance+trade%2Doff%2E+Proceedings+of+the+National+Academy+of+Sciences+of+the)
7. Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020. 29 — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) [find](https://scholar.google.com/scholar?q=Lucas+Beyer%2C+Olivier+J+H%C3%A9naff%2C+Alexander+Kolesnikov%2C+Xiaohua+Zhai%2C+and+A%C3%A4ron+van+den+Oord%2E+Are+we+done+with+imagenet%3F+arXiv+preprint+arXiv%3A2006%2E07159%2C+2020%2E+29+%E2%80%94+%5BarXiv%3A2006%2E07159%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
9. Sebastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 28811–28822. Curran Associates, Inc., 2021. 4 [find](https://scholar.google.com/scholar?q=Sebastien+Bubeck+and+Mark+Sellke%2E+A+universal+law+of+robustness+via+isoperimetry%2E+In+M%2E+Ranzato%2C+A%2E+Beygelzimer%2C+Y%2E+Dauphin%2C+P%2ES%2E+Liang%2C+and+J%2E+Wortman+Vaughan%2C+editors%2C+Advances+in+Neural+Information)
10. Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset), 2022. 5 https://gauss-centre.eu [↗](https://github.com/kakaobrain/coyo-dataset)
11. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag… [find](https://scholar.google.com/scholar?q=Tianlong+Chen%2C+Jonathan+Frankle%2C+Shiyu+Chang%2C+Sijia+Liu%2C+Yang+Zhang%2C+Michael+Carbin%2C+and+Zhangyang+Wang%2E+The+lottery+tickets+hypothesis+for+supervised+and+self%2Dsupervised+pre%2Dtraining+in+computer+visi)
12. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 4 [our card](https://md.research.elamur.ai/free/en/papers/chen-2022-pali-jointly-scaled-multilingual-language/)
13. Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), 2017. [https://ieeexplore.ieee.org/abstract/document/7891544](https://ieeexplore.ieee.org/abstract/document/7891544). 9 [↗](https://ieeexplore.ieee.org/abstract/document/7891544)
14. Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014. [https://openaccess.thecvf.com/content_cvpr_2014/html/Cimpoi_Describing_Textures_](https://openaccess.thecvf.com/content_cvpr_2014/html/Cimpoi_Describing_Textures_) in_2014_CVPR_paper.html. 9 [↗](https://openaccess.thecvf.com/content_cvpr_2014/html/Cimpoi_Describing_Textures_)
15. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 248–255, June 2009. 2, 6, 8, 9, 28 [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Proc%2E+IEEE+Conf%2E+Computer+Vision+and+Pattern+Recognition%2C+pages+248%E2%80%93255%2C+June+2009%2E+)
16. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. 6 [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2C+2022%2E+6)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
18. Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). arXiv preprint arXiv:2205.01397, 2022. 4 — [arXiv:2205.01397](https://arxiv.org/abs/2205.01397) [find](https://scholar.google.com/scholar?q=Alex+Fang%2C+Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Yuhao+Wan%2C+Vaishaal+Shankar%2C+Achal+Dave%2C+and+Ludwig+Schmidt%2E+Data+determines+distributional+robustness+in+contrastive+language+image+pre%2Dtraining+%28clip%29%2E)
19. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022. 21 — [arXiv:2211.07636](https://arxiv.org/abs/2211.07636) [find](https://scholar.google.com/scholar?q=Yuxin+Fang%2C+Wen+Wang%2C+Binhui+Xie%2C+Quan+Sun%2C+Ledell+Wu%2C+Xinggang+Wang%2C+Tiejun+Huang%2C+Xinlong+Wang%2C+and+Yue+Cao%2E+Eva%3A+Exploring+the+limits+of+masked+visual+representation+learning+at+scale%2E+arXiv+prepri)
20. Stephan Graf and Olaf Mextorf. Just: Large-scale multi-tier storage infrastructure at the jülich supercomputing centre. Journal of large-scale research facilities JLSRF, 7:180, 2021. 13 [find](https://scholar.google.com/scholar?q=Stephan+Graf+and+Olaf+Mextorf%2E+Just%3A+Large%2Dscale+multi%2Dtier+storage+infrastructure+at+the+j%C3%BClich+supercomputing+centre%2E+Journal+of+large%2Dscale+research+facilities+JLSRF%2C+7%3A180%2C+2021%2E+13)
21. Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. https://arxiv.org/abs/1709.00029. 9 — [arXiv:1709.00029](https://arxiv.org/abs/1709.00029) [↗](https://arxiv.org/abs/1709.00029)
22. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. International Conference on… — [arXiv:2006.16241](https://arxiv.org/abs/2006.16241) [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Steven+Basart%2C+Norman+Mu%2C+Saurav+Kadavath%2C+Frank+Wang%2C+Evan+Dorundo%2C+Rahul+Desai%2C+Tyler+Zhu%2C+Samyak+Parajuli%2C+Mike+Guo%2C+Dawn+Song%2C+Jacob+Steinhardt%2C+and+Justin+Gilmer%2E+The+many+faces+of)
23. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1903.12261. 6, 7, 9 [↗](https://arxiv.org/abs/1903.12261)
24. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. Conference on Computer Vision and Pattern Recognition (CVPR), 2021. https://arxiv.org/abs/1907.07174. 3, 6, 7, 9, 29 — [arXiv:1907.07174](https://arxiv.org/abs/1907.07174) [↗](https://arxiv.org/abs/1907.07174)
25. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autor… [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
26. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. 3 [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
27. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022. 5, 21 — [arXiv:2210.02303](https://arxiv.org/abs/2210.02303) [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+William+Chan%2C+Chitwan+Saharia%2C+Jay+Whang%2C+Ruiqi+Gao%2C+Alexey+Gritsenko%2C+Diederik+P+Kingma%2C+Ben+Poole%2C+Mohammad+Norouzi%2C+David+J+Fleet%2C+et+al%2E+Imagen+video%3A+High+definition+video+generation)
28. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. 1, 2, 4, 6, 12 [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
29. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia, July 2018. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+L)
30. Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17980–17989, June 2022. 5 [find](https://scholar.google.com/scholar?q=Xiaowei+Hu%2C+Zhe+Gan%2C+Jianfeng+Wang%2C+Zhengyuan+Yang%2C+Zicheng+Liu%2C+Yumao+Lu%2C+and+Lijuan+Wang%2E+Scaling+up+vision%2Dlanguage+pre%2Dtraining+for+image+captioning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+C)
31. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. arXiv preprint arXiv:2208.05592, 2022. 9, 10, 23, 27, 29 — [arXiv:2208.05592](https://arxiv.org/abs/2208.05592) [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Samir+Yitzhak+Gadre%2C+Shuran+Song%2C+Hannaneh+Hajishirzi%2C+Simon+Kornblith%2C+Ali+Farhadi%2C+and+Ludwig+Schmidt%2E+Patching+open%2Dvocabulary+models+by+interpolating+weights%2E+a)
32. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. 2, 6, 12, 19 [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Ross+Wightman%2C+Cade+Gordon%2C+Nicholas+Carlini%2C+Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Hongseok+Namkoong%2C+John+Miller%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+and+Ludwi)
33. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021. 1, 4 [our card](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
34. Juelich Supercomputing Center. JUWELS Booster Supercomputer, 2020. [https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html\#](https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html\#) hardware-configuration-of-the-system-name-booster-module. 6, 13, 19 [↗](https://apps.fz-juelich.de/jsc/hps/juwels/configuration.html\#)
35. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 1, 2, 4, 11, 12 [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
36. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015. 7 [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
37. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 9 [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E+9)
38. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 491–507, Cham… [find](https://scholar.google.com/scholar?q=Alexander+Kolesnikov%2C+Lucas+Beyer%2C+Xiaohua+Zhai%2C+Joan+Puigcerver%2C+Jessica+Yung%2C+Sylvain+Gelly%2C+and+Neil+Houlsby%2E+Big+transfer+%28bit%29%3A+General+visual+representation+learning%2E+In+Andrea+Vedaldi%2C+Horst+Bi)
39. Skanda Koppula, Yazhe Li, Evan Shelhamer, Andrew Jaegle, Nikhil Parthasarathy, Relja Arandjelovic, João Carreira, and Olivier Hénaff. Where should i spend my flops? efficiency evaluations of visual pre-training methods. arXiv preprint arXiv:2209.15589, 2022. 4, 12 — [arXiv:2209.15589](https://arxiv.org/abs/2209.15589) [find](https://scholar.google.com/scholar?q=Skanda+Koppula%2C+Yazhe+Li%2C+Evan+Shelhamer%2C+Andrew+Jaegle%2C+Nikhil+Parthasarathy%2C+Relja+Arandjelovic%2C+Jo%C3%A3o+Carreira%2C+and+Olivier+H%C3%A9naff%2E+Where+should+i+spend+my+flops%3F+efficiency+evaluations+of+visual+pr)
40. Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. https://arxiv.org/abs/ 1805.08974. 9 [↗](https://arxiv.org/abs/)
41. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In International Conference on Computer Vision Workshops (ICML), 2013. [https://www.cv-foundation.org/openaccess/content_iccv_workshops_2013/W19/html/](https://www.cv-foundation.org/openaccess/content_iccv_workshops_2013/W19/html/) Krause_3D_Object_Representations_… [↗](https://www.cv-foundation.org/openaccess/content_iccv_workshops_2013/W19/html/)
42. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32–73, 2… [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
43. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. 8, 9, 28 [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+2009%2E+8%2C+9%2C+28)
44. Yann LeCun. The mnist database of handwritten digits, 1998. [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/). 9 [↗](http://yann.lecun.com/exdb/mnist/)
45. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020. 6 [our card](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/)
46. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 2, 5, 7 [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
47. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 6, 10 [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E+6%2C+10)
48. John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning,… [find](https://scholar.google.com/scholar?q=John+P+Miller%2C+Rohan+Taori%2C+Aditi+Raghunathan%2C+Shiori+Sagawa%2C+Pang+Wei+Koh%2C+Vaishaal+Shankar%2C+Percy+Liang%2C+Yair+Carmon%2C+and+Ludwig+Schmidt%2E+Accuracy+on+the+line%3A+on+the+strong+correlation+between+out%2D)
49. Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint arXiv:2206.02770, 2022. 4, 5, 21 — [arXiv:2206.02770](https://arxiv.org/abs/2206.02770) [find](https://scholar.google.com/scholar?q=Basil+Mustafa%2C+Carlos+Riquelme%2C+Joan+Puigcerver%2C+Rodolphe+Jenatton%2C+and+Neil+Houlsby%2E+Multimodal+contrastive+learning+with+limoe%3A+the+language%2Dimage+mixture+of+experts%2E+arXiv+preprint+arXiv%3A2206%2E02770)
50. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPS) Workshops, 2011. [https://storage.googleapis.com/](https://storage.googleapis.com/) pub-tools-public-publication-data/pdf… [↗](https://storage.googleapis.com/)
51. Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. arXiv preprint arXiv:2208.05516, 2022. 4 — [arXiv:2208.05516](https://arxiv.org/abs/2208.05516) [find](https://scholar.google.com/scholar?q=Thao+Nguyen%2C+Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Sewoong+Oh%2C+and+Ludwig+Schmidt%2E+Quality+not+quantity%3A+On+the+interaction+between+dataset+design+and+robustness+of+clip%2E+arXiv+preprint+arXiv%3A2208%2E05516)
52. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 19 [our card](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
53. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 6 [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
54. Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050, 2021. 1, 4, 5, 11 [find](https://scholar.google.com/scholar?q=Hieu+Pham%2C+Zihang+Dai%2C+Golnaz+Ghiasi%2C+Hanxiao+Liu%2C+Adams+Wei+Yu%2C+Minh%2DThang+Luong%2C+Mingxing+Tan%2C+and+Quoc+V+Le%2E+Combined+scaling+for+zero%2Dshot+transfer+learning%2E+arXiv+preprint+arXiv%3A2111%2E10050%2C+2021%2E)
55. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 20… [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
56. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. Technical report, Tech. Rep., Technical report, OpenAI, 2022. 1 [our card](https://md.research.elamur.ai/free/en/papers/radford-2022-robust-speech-recognition-large-scale/)
57. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. 1 [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
58. Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021. 4 [find](https://scholar.google.com/scholar?q=Vinay+Venkatesh+Ramasesh%2C+Aitor+Lewkowycz%2C+and+Ethan+Dyer%2E+Effect+of+scale+on+catastrophic+forgetting+in+neural+networks%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E+4)
59. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. 1 [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2022-hierarchical-text-conditional-image-generation/)
60. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning (ICML), 2019. https://arxiv.org/abs/1902.10811. 3, 29 [↗](https://arxiv.org/abs/1902.10811)
61. Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34, 2021. 1, 2, 4, 11 [find](https://scholar.google.com/scholar?q=Carlos+Riquelme%2C+Joan+Puigcerver%2C+Basil+Mustafa%2C+Maxim+Neumann%2C+Rodolphe+Jenatton%2C+Andr%C3%A9+Susano+Pinto%2C+Daniel+Keysers%2C+and+Neil+Houlsby%2E+Scaling+vision+with+sparse+mixture+of+experts%2E+Advances+in+Neur)
62. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022. 1, 5, 21 [find](https://scholar.google.com/scholar?q=Robin+Rombach%2C+Andreas+Blattmann%2C+Dominik+Lorenz%2C+Patrick+Esser%2C+and+Bj%C3%B6rn+Ommer%2E+High%2Dresolution+image+synthesis+with+latent+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vi)
63. Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9075–90… [find](https://scholar.google.com/scholar?q=Jonathan+S+Rosenfeld%2C+Jonathan+Frankle%2C+Michael+Carbin%2C+and+Nir+Shavit%2E+On+the+predictability+of+pruning+across+scales%2E+In+Marina+Meila+and+Tong+Zhang%2C+editors%2C+Proceedings+of+the+38th+International+C)
64. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-… [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)
65. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 5, 12, 20 [our card](https://md.research.elamur.ai/free/en/papers/schuhmann-2021-laion-400m-open-dataset-clip/)
66. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, Melbou… [our card](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
67. Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486, 2022. 12 [our card](https://md.research.elamur.ai/free/en/papers/sorscher-2022-neural-scaling-laws-beating-power/)
68. Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In International Joint Conference on Neural Networks (IJCNN), 2011. [https://ieeexplore.ieee.org/document/6033395](https://ieeexplore.ieee.org/document/6033395). 9 [↗](https://ieeexplore.ieee.org/document/6033395)
69. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017. 3 [find](https://scholar.google.com/scholar?q=Chen+Sun%2C+Abhinav+Shrivastava%2C+Saurabh+Singh%2C+and+Abhinav+Gupta%2E+Revisiting+unreasonable+effectiveness+of+data+in+deep+learning+era%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vis)
70. Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583–18599, 2020. 2, 4, 12 [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Nicholas+Carlini%2C+Benjamin+Recht%2C+and+Ludwig+Schmidt%2E+Measuring+robustness+to+natural+distribution+shifts+in+image+classification%2E+Advances+in+Neural+Informa)
71. Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022. 4 [our card](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)
72. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In International Conference on Learning Representations, 2021. 1, 2, 4, 11 [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Jinfeng+Rao%2C+William+Fedus%2C+Samira+Abnar%2C+Hyung+Won+Chung%2C+Sharan+Narang%2C+Dani+Yogatama%2C+Ashish+Vaswani%2C+and+Donald+Metzler%2E+Scale+efficiently%3A+Insights+from+pretraining+and+)
73. Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. 5 [find](https://scholar.google.com/scholar?q=Bart+Thomee%2C+David+A+Shamma%2C+Gerald+Friedland%2C+Benjamin+Elizalde%2C+Karl+Ni%2C+Douglas+Poland%2C+Damian+Borth%2C+and+Li%2DJia+Li%2E+Yfcc100m%3A+The+new+data+in+multimedia+research%2E+Communications+of+the+ACM%2C+59%282%29%3A)
74. Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1905.13549. 3, 6, 7, 9, 29 — [arXiv:1905.13549](https://arxiv.org/abs/1905.13549) [↗](https://arxiv.org/abs/1905.13549)
75. Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint arXiv:2211.05778, 2022. 21 — [arXiv:2211.05778](https://arxiv.org/abs/2211.05778) [find](https://scholar.google.com/scholar?q=Wenhai+Wang%2C+Jifeng+Dai%2C+Zhe+Chen%2C+Zhenhang+Huang%2C+Zhiqi+Li%2C+Xizhou+Zhu%2C+Xiaowei+Hu%2C+Tong+Lu%2C+Lewei+Lu%2C+Hongsheng+Li%2C+et+al%2E+Internimage%3A+Exploring+large%2Dscale+vision+foundation+models+with+deformable)
76. Ross Wightman. Pytorch image models. [https://github.com/rwightman/pytorch-image-models](https://github.com/rwightman/pytorch-image-models), 2019. 9 [↗](https://github.com/rwightman/pytorch-image-models)
77. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio… [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Gabriel+Ilharco%2C+Jong+Wook+Kim%2C+Mike+Li%2C+Simon+Kornblith%2C+Rebecca+Roelofs%2C+Raphael+Gontijo+Lopes%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+Hongseok+Namkoong%2C+et+al%2E+Robust+fine%2Dtuning+of+z)
78. Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), 2016. [https://link.springer.com/article/10.1007/s11263-014-0748-y](https://link.springer.com/article/10.1007/s11263-014-0748-y). 9 — [doi:10.1007/s11263-014-0748-y](https://doi.org/10.1007/s11263-014-0748-y) [↗](https://link.springer.com/article/10.1007/s11263-014-0748-y)
79. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 7 [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of+the+A)
80. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 4, 11 [our card](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/)
81. Christoph Zauner. Implementation and benchmarking of perceptual image hash functions. 2010. 6, 21, 22 [find](https://scholar.google.com/scholar?q=Christoph+Zauner%2E+Implementation+and+benchmarking+of+perceptual+image+hash+functions%2E+2010%2E+6%2C+21%2C+22)
82. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022. 1, 2, 4, 11, 12 [find](https://scholar.google.com/scholar?q=Xiaohua+Zhai%2C+Alexander+Kolesnikov%2C+Neil+Houlsby%2C+and+Lucas+Beyer%2E+Scaling+vision+transformers%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+12104%E2%80%9312113%2C)
83. Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867,… — [arXiv:1910.04867](https://arxiv.org/abs/1910.04867) [find](https://scholar.google.com/scholar?q=Xiaohua+Zhai%2C+Joan+Puigcerver%2C+Alexander+Kolesnikov%2C+Pierre+Ruyssen%2C+Carlos+Riquelme%2C+Mario+Lucic%2C+Josip+Djolonga%2C+Andre+Susano+Pinto%2C+Maxim+Neumann%2C+Alexey+Dosovitskiy%2C+et+al%2E+A+large%2Dscale+study+of+)
84. Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022. 4, 5, 6, 11, 12, 21 [find](https://scholar.google.com/scholar?q=Xiaohua+Zhai%2C+Xiao+Wang%2C+Basil+Mustafa%2C+Andreas+Steiner%2C+Daniel+Keysers%2C+Alexander+Kolesnikov%2C+and+Lucas+Beyer%2E+LiT%3A+Zero%2Dshot+transfer+with+locked%2Dimage+text+tuning%2E+In+Proceedings+of+the+IEEE%2FCVF+Co)

*Updated 23 October 2026*